2023-01-11T20:41:05.2281975Z Requested labels: linux.2xlarge
2023-01-11T20:41:05.2282094Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/91627/merge
2023-01-11T20:41:05.2282218Z Reusable workflow chain:
2023-01-11T20:41:05.2282267Z pytorch/pytorch/.github/workflows/pull.yml@refs/pull/91627/merge (57fc38f02f250896a12b32cfa200a6105a03d09c)
2023-01-11T20:41:05.2282314Z -> pytorch/pytorch/.github/workflows/_linux-test.yml@refs/pull/91627/merge (57fc38f02f250896a12b32cfa200a6105a03d09c)
2023-01-11T20:41:05.2282343Z Waiting for a runner to pick up this job...
2023-01-11T20:41:05.8456919Z Job is about to start running on the runner: i-01f7d826cf45480d0 (organization)
2023-01-11T20:41:10.2429268Z Current runner version: '2.300.2'
2023-01-11T20:41:10.2433723Z Runner name: 'i-01f7d826cf45480d0'
2023-01-11T20:41:10.2434050Z Runner group name: 'Default'
2023-01-11T20:41:10.2434445Z Machine name: 'ip-10-0-1-8'
2023-01-11T20:41:10.2435775Z ##[group]GITHUB_TOKEN Permissions
2023-01-11T20:41:10.2436372Z Actions: read
2023-01-11T20:41:10.2436533Z Checks: read
2023-01-11T20:41:10.2436705Z Contents: read
2023-01-11T20:41:10.2436885Z Deployments: read
2023-01-11T20:41:10.2437051Z Discussions: read
2023-01-11T20:41:10.2437229Z Issues: read
2023-01-11T20:41:10.2437400Z Metadata: read
2023-01-11T20:41:10.2437558Z Packages: read
2023-01-11T20:41:10.2437726Z Pages: read
2023-01-11T20:41:10.2437899Z PullRequests: read
2023-01-11T20:41:10.2438119Z RepositoryProjects: read
2023-01-11T20:41:10.2438316Z SecurityEvents: read
2023-01-11T20:41:10.2438492Z Statuses: read
2023-01-11T20:41:10.2438652Z ##[endgroup]
2023-01-11T20:41:10.2441433Z Secret source: None
2023-01-11T20:41:10.2441788Z Prepare workflow directory
2023-01-11T20:41:10.4661024Z Prepare all required actions
2023-01-11T20:41:10.4823026Z Getting action download info
2023-01-11T20:41:10.8066438Z Download action repository 'pytorch/test-infra@main' (SHA:2c225610d00fb13c04fcd60389d3e4d8326167c3)
2023-01-11T20:41:11.0685409Z Download action repository 'pytorch/pytorch@master' (SHA:c5836153f5332ca83d5cacde38f2829a4d54793e)
2023-01-11T20:41:13.6491289Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2023-01-11T20:41:13.9137254Z Getting action download info
2023-01-11T20:41:14.1068639Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076)
2023-01-11T20:41:14.2765726Z Getting action download info
2023-01-11T20:41:14.4799161Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2023-01-11T20:41:14.6284173Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml
2023-01-11T20:41:14.6285768Z ##[group] Inputs
2023-01-11T20:41:14.6286047Z   build-environment: linux-focal-py3.7-gcc7
2023-01-11T20:41:14.6286814Z   test-matrix: { include: [
  { config: "default", shard: 1, num_shards: 2, runner: "linux.2xlarge" },
  { config: "default", shard: 2, num_shards: 2, runner: "linux.2xlarge" },
  { config: "distributed", shard: 1, num_shards: 2, runner: "linux.2xlarge" },
  { config: "distributed", shard: 2, num_shards: 2, runner: "linux.2xlarge" },
  { config: "functorch", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
  { config: "docs_test", shard: 1, num_shards: 1,  runner: "linux.2xlarge" },
  { config: "jit_legacy", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
  { config: "backwards_compat", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
]}

2023-01-11T20:41:14.6287653Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T20:41:14.6287984Z   sync-tag: 
2023-01-11T20:41:14.6288637Z   timeout-minutes: 240
2023-01-11T20:41:14.6288817Z   use-gha: 
2023-01-11T20:41:14.6289007Z ##[endgroup]
2023-01-11T20:41:14.6289452Z Complete job name: linux-focal-py3.7-gcc7 / test (distributed, 2, 2, linux.2xlarge)
2023-01-11T20:41:14.6946880Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2023-01-11T20:41:14.6947155Z with:
2023-01-11T20:41:14.6947636Z   github-secret: ***
2023-01-11T20:41:14.6947964Z   instructions: All testing is done inside the container, to start an interactive session run:
  docker exec -it $(docker container ps --format '{{.ID}}') bash

2023-01-11T20:41:14.6948299Z   activate-with-label: false
2023-01-11T20:41:14.6948494Z   label: with-ssh
2023-01-11T20:41:14.6948683Z   remove-existing-keys: true
2023-01-11T20:41:14.6948854Z env:
2023-01-11T20:41:14.6949023Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:41:14.6949207Z ##[endgroup]
2023-01-11T20:41:15.2422963Z Grabbing public ssh keys from https://github.com/LucaLumetti.keys
2023-01-11T20:41:15.3150768Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh
2023-01-11T20:41:15.3163193Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys
2023-01-11T20:41:15.3187507Z Login using: ssh ec2-user@ec2-34-232-109-219.compute-1.amazonaws.com
2023-01-11T20:41:15.3187893Z All testing is done inside the container, to start an interactive session run:
2023-01-11T20:41:15.3188248Z   docker exec -it $(docker container ps --format '{{.ID}}') bash
2023-01-11T20:41:15.3395371Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2023-01-11T20:41:15.3395633Z with:
2023-01-11T20:41:15.3395809Z   submodules: recursive
2023-01-11T20:41:15.3395981Z   fetch-depth: 0
2023-01-11T20:41:15.3396144Z env:
2023-01-11T20:41:15.3396316Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:41:15.3396489Z ##[endgroup]
2023-01-11T20:41:15.3597628Z ##[group]Run retry () {
2023-01-11T20:41:15.3597878Z retry () {
2023-01-11T20:41:15.3598104Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2023-01-11T20:41:15.3598311Z }
2023-01-11T20:41:15.3598496Z echo "${GITHUB_WORKSPACE}"
2023-01-11T20:41:15.3598713Z if [ -z "${NO_SUDO}" ]; then
2023-01-11T20:41:15.3598923Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2023-01-11T20:41:15.3599127Z else
2023-01-11T20:41:15.3599322Z   retry rm -rf "${GITHUB_WORKSPACE}"
2023-01-11T20:41:15.3599500Z fi
2023-01-11T20:41:15.3599713Z mkdir "${GITHUB_WORKSPACE}"
2023-01-11T20:41:15.3615032Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T20:41:15.3615265Z env:
2023-01-11T20:41:15.3615443Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:41:15.3615628Z   NO_SUDO: 
2023-01-11T20:41:15.3615784Z ##[endgroup]
2023-01-11T20:41:15.3706091Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T20:41:17.4164509Z ##[group]Run malfet/checkout@silent-checkout
2023-01-11T20:41:17.4164748Z with:
2023-01-11T20:41:17.4164954Z   ref: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T20:41:17.4165180Z   fetch-depth: 0
2023-01-11T20:41:17.4165349Z   submodules: recursive
2023-01-11T20:41:17.4165537Z   quiet-checkout: true
2023-01-11T20:41:17.4165738Z   repository: pytorch/pytorch
2023-01-11T20:41:17.4166058Z   token: ***
2023-01-11T20:41:17.4166233Z   ssh-strict: true
2023-01-11T20:41:17.4166428Z   persist-credentials: true
2023-01-11T20:41:17.4166618Z   clean: true
2023-01-11T20:41:17.4166777Z   lfs: false
2023-01-11T20:41:17.4166958Z   set-safe-directory: true
2023-01-11T20:41:17.4167139Z env:
2023-01-11T20:41:17.4167295Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:41:17.4167483Z ##[endgroup]
2023-01-11T20:41:17.5434665Z Syncing repository: pytorch/pytorch
2023-01-11T20:41:17.5436001Z ##[group]Getting Git version info
2023-01-11T20:41:17.5436421Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2023-01-11T20:41:17.5436852Z [command]/usr/bin/git version
2023-01-11T20:41:17.5437043Z git version 2.38.1
2023-01-11T20:41:17.5437593Z ##[endgroup]
2023-01-11T20:41:17.5448314Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/9fc50926-9697-4b0b-88d4-9bb92f867657' before making global git config changes
2023-01-11T20:41:17.5448711Z Adding repository directory to the temporary git global config as a safe directory
2023-01-11T20:41:17.5452352Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T20:41:17.5489818Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2023-01-11T20:41:17.5494209Z ##[group]Initializing the repository
2023-01-11T20:41:17.5496933Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T20:41:17.5525118Z hint: Using 'master' as the name for the initial branch. This default branch name
2023-01-11T20:41:17.5525669Z hint: is subject to change. To configure the initial branch name to use in all
2023-01-11T20:41:17.5526173Z hint: of your new repositories, which will suppress this warning, call:
2023-01-11T20:41:17.5526684Z hint: 
2023-01-11T20:41:17.5527191Z hint: 	git config --global init.defaultBranch <name>
2023-01-11T20:41:17.5527467Z hint: 
2023-01-11T20:41:17.5527737Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2023-01-11T20:41:17.5528098Z hint: 'development'. The just-created branch can be renamed via this command:
2023-01-11T20:41:17.5528332Z hint: 
2023-01-11T20:41:17.5528601Z hint: 	git branch -m <name>
2023-01-11T20:41:17.5529320Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2023-01-11T20:41:17.5535475Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2023-01-11T20:41:17.5563004Z ##[endgroup]
2023-01-11T20:41:17.5563600Z ##[group]Disabling automatic garbage collection
2023-01-11T20:41:17.5566296Z [command]/usr/bin/git config --local gc.auto 0
2023-01-11T20:41:17.5591939Z ##[endgroup]
2023-01-11T20:41:17.5592746Z ##[group]Setting up auth
2023-01-11T20:41:17.5598372Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2023-01-11T20:41:17.5626911Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2023-01-11T20:41:17.5868366Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2023-01-11T20:41:17.5896408Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2023-01-11T20:41:17.6133251Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2023-01-11T20:41:17.6173777Z ##[endgroup]
2023-01-11T20:41:17.6174249Z ##[group]Fetching the repository
2023-01-11T20:41:17.6180029Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2023-01-11T20:42:10.9988693Z [command]/usr/bin/git rev-parse --verify --quiet 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e^{object}
2023-01-11T20:42:11.0013569Z 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T20:42:11.0018850Z ##[endgroup]
2023-01-11T20:42:11.0019357Z ##[group]Determining the checkout info
2023-01-11T20:42:11.0019973Z ##[endgroup]
2023-01-11T20:42:11.0020510Z ##[group]Checking out the ref
2023-01-11T20:42:11.0023732Z [command]/usr/bin/git checkout --quiet --force 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T20:42:12.2865692Z ##[endgroup]
2023-01-11T20:42:12.2866186Z ##[group]Setting up auth for fetching submodules
2023-01-11T20:42:12.2871546Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2023-01-11T20:42:12.2917705Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2023-01-11T20:42:12.2948378Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2023-01-11T20:42:12.2980947Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2023-01-11T20:42:12.3006511Z ##[endgroup]
2023-01-11T20:42:12.3007126Z ##[group]Fetching submodules
2023-01-11T20:42:12.3011123Z [command]/usr/bin/git submodule sync --recursive
2023-01-11T20:42:12.3264459Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2023-01-11T20:42:12.3509945Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2023-01-11T20:42:12.3510747Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2023-01-11T20:42:12.3512789Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2023-01-11T20:42:12.3514895Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2023-01-11T20:42:12.3517173Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2023-01-11T20:42:12.3519544Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2023-01-11T20:42:12.3521895Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2023-01-11T20:42:12.3524415Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2023-01-11T20:42:12.3527139Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2023-01-11T20:42:12.3529805Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2023-01-11T20:42:12.3532766Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2023-01-11T20:42:12.3535731Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2023-01-11T20:42:12.3538761Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2023-01-11T20:42:12.3541805Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2023-01-11T20:42:12.3545087Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2023-01-11T20:42:12.3548305Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2023-01-11T20:42:12.3551736Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2023-01-11T20:42:12.3555362Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2023-01-11T20:42:12.3559613Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2023-01-11T20:42:12.3563588Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2023-01-11T20:42:12.3567193Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2023-01-11T20:42:12.3571336Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2023-01-11T20:42:12.3575494Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2023-01-11T20:42:12.3579585Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2023-01-11T20:42:12.3583823Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2023-01-11T20:42:12.3588189Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2023-01-11T20:42:12.3592585Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2023-01-11T20:42:12.3597061Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2023-01-11T20:42:12.3601732Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2023-01-11T20:42:12.3606415Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2023-01-11T20:42:12.3612288Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2023-01-11T20:42:12.3617560Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2023-01-11T20:42:12.3622607Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2023-01-11T20:42:12.3627666Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2023-01-11T20:42:12.3632969Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2023-01-11T20:42:12.3638289Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2023-01-11T20:42:12.3643695Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2023-01-11T20:42:12.3649150Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2023-01-11T20:42:12.3655135Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2023-01-11T20:42:12.3660971Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2023-01-11T20:42:12.3666820Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2023-01-11T20:42:12.3690795Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2023-01-11T20:42:12.6746259Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2023-01-11T20:42:12.8967734Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2023-01-11T20:42:13.1877606Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2023-01-11T20:42:13.5103260Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2023-01-11T20:42:13.7999100Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2023-01-11T20:42:15.8942322Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2023-01-11T20:42:22.3052770Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2023-01-11T20:42:22.7458973Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2023-01-11T20:42:23.2562031Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2023-01-11T20:42:24.6708392Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2023-01-11T20:42:25.7419797Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2023-01-11T20:42:26.8723470Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2023-01-11T20:42:32.0915615Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2023-01-11T20:42:32.8093977Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2023-01-11T20:42:34.2867196Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2023-01-11T20:42:35.5190538Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2023-01-11T20:42:35.7218116Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2023-01-11T20:42:36.1888871Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2023-01-11T20:42:36.5047546Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2023-01-11T20:42:37.5364652Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2023-01-11T20:42:37.9918851Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2023-01-11T20:42:38.2102436Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2023-01-11T20:42:38.5095685Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2023-01-11T20:42:39.8777339Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2023-01-11T20:42:40.2232736Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2023-01-11T20:42:40.5907348Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2023-01-11T20:42:46.2960179Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2023-01-11T20:42:48.2664790Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2023-01-11T20:42:48.7213630Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2023-01-11T20:42:48.9494744Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2023-01-11T20:42:54.4875364Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2023-01-11T20:42:54.6940361Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2023-01-11T20:42:54.9353776Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2023-01-11T20:42:55.7558549Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2023-01-11T20:42:55.9783269Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2023-01-11T20:42:56.3128064Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2023-01-11T20:42:56.5988707Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2023-01-11T20:42:57.1646840Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2023-01-11T20:42:58.9720391Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2023-01-11T20:42:59.4587884Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2023-01-11T20:43:01.7422167Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2023-01-11T20:43:01.7518069Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2023-01-11T20:43:01.7591139Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2023-01-11T20:43:01.7791901Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2023-01-11T20:43:01.7990700Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2023-01-11T20:43:01.8307469Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2023-01-11T20:43:02.3780442Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2023-01-11T20:43:02.3969882Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2023-01-11T20:43:02.4876061Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c'
2023-01-11T20:43:02.5187261Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2023-01-11T20:43:02.7816981Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3'
2023-01-11T20:43:03.1524390Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a'
2023-01-11T20:43:03.3743257Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2023-01-11T20:43:03.4160238Z Submodule path 'third_party/fbgemm': checked out '80d64206c07879fd4683be66873de7cefa1a0a71'
2023-01-11T20:43:03.4174393Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2023-01-11T20:43:03.4176710Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T20:43:03.4179188Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2023-01-11T20:43:03.4181842Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T20:43:03.4202819Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2023-01-11T20:43:04.4531403Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2023-01-11T20:43:05.2878731Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2023-01-11T20:43:06.3781023Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2023-01-11T20:43:06.7091053Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2023-01-11T20:43:06.8006680Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2023-01-11T20:43:06.8538628Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2023-01-11T20:43:06.8632184Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5'
2023-01-11T20:43:06.9403793Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2023-01-11T20:43:06.9732891Z Submodule path 'third_party/fmt': checked out '7bdf0628b1276379886c7f6dda2cef2b3b374f0b'
2023-01-11T20:43:06.9813285Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2023-01-11T20:43:07.0171985Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2023-01-11T20:43:07.0385985Z Submodule path 'third_party/gloo': checked out '4a5e339b764261d20fc409071dc7a8b8989aa195'
2023-01-11T20:43:07.0798151Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2023-01-11T20:43:07.0902825Z Submodule path 'third_party/ideep': checked out 'e533c771a1e75a1c225c14b2261eefa62681d9e6'
2023-01-11T20:43:07.0915120Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2023-01-11T20:43:07.0936093Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2023-01-11T20:43:15.9459913Z Submodule path 'third_party/ideep/mkl-dnn': checked out '404ad76ee633c939d705eb583ffe50a806969d5e'
2023-01-11T20:43:15.9476207Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T20:43:15.9499012Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2023-01-11T20:43:24.3374060Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out 'fbec3e25a559ee252022ae066817b204e106a6ba'
2023-01-11T20:43:24.3466366Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2023-01-11T20:43:24.3595455Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2023-01-11T20:43:24.4439723Z Submodule path 'third_party/kineto': checked out '6c1629809068efd78a8d56b4aa479c7ec49ae562'
2023-01-11T20:43:24.4455441Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T20:43:24.4457500Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T20:43:24.4480263Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2023-01-11T20:43:25.6740963Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2023-01-11T20:43:26.7443462Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2023-01-11T20:43:26.7935251Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2023-01-11T20:43:26.8126947Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52'
2023-01-11T20:43:26.8253671Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2023-01-11T20:43:26.9226092Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2023-01-11T20:43:27.1467800Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b'
2023-01-11T20:43:27.1494497Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2023-01-11T20:43:27.1496280Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2023-01-11T20:43:27.1519416Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2023-01-11T20:43:27.5399439Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2023-01-11T20:43:28.4128588Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2023-01-11T20:43:28.4409446Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8'
2023-01-11T20:43:28.4543135Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2023-01-11T20:43:28.4556358Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T20:43:28.4577413Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2023-01-11T20:43:30.4612130Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2023-01-11T20:43:30.4631480Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T20:43:30.4633454Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T20:43:30.4657218Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2023-01-11T20:43:30.8558717Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2023-01-11T20:43:31.7103382Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2023-01-11T20:43:31.7708159Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2023-01-11T20:43:31.7720260Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T20:43:31.7742651Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2023-01-11T20:43:31.9904435Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2023-01-11T20:43:31.9985666Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2023-01-11T20:43:32.2324455Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2023-01-11T20:43:32.2342946Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2023-01-11T20:43:32.2345223Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2023-01-11T20:43:32.2368447Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2023-01-11T20:43:32.6636222Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2023-01-11T20:43:33.6755021Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2023-01-11T20:43:33.7374308Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2023-01-11T20:43:33.7449550Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2023-01-11T20:43:33.7542444Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2023-01-11T20:43:33.7840894Z Submodule path 'third_party/pybind11': checked out '80dc998efced8ceb2be59756668a7e90e8bef917'
2023-01-11T20:43:33.7922808Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2023-01-11T20:43:33.8184807Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2023-01-11T20:43:33.8267821Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2023-01-11T20:43:33.8663025Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2023-01-11T20:43:33.9641652Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2023-01-11T20:43:33.9873883Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2023-01-11T20:43:33.9887061Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2023-01-11T20:43:33.9888965Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2023-01-11T20:43:33.9891240Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2023-01-11T20:43:33.9893535Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T20:43:33.9916011Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2023-01-11T20:43:35.0569832Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2023-01-11T20:43:35.3341353Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2023-01-11T20:43:36.6251875Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2023-01-11T20:43:37.6232839Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2023-01-11T20:43:37.6364893Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2023-01-11T20:43:37.6932851Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2023-01-11T20:43:37.7178930Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2023-01-11T20:43:37.7191970Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T20:43:37.7218427Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2023-01-11T20:43:37.9339355Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2023-01-11T20:43:38.0577158Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2023-01-11T20:43:38.0603681Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2023-01-11T20:43:38.0859728Z Entering 'android/libs/fbjni'
2023-01-11T20:43:38.0894625Z Entering 'third_party/FP16'
2023-01-11T20:43:38.0928874Z Entering 'third_party/FXdiv'
2023-01-11T20:43:38.0962722Z Entering 'third_party/NNPACK'
2023-01-11T20:43:38.0997820Z Entering 'third_party/QNNPACK'
2023-01-11T20:43:38.1032348Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T20:43:38.1067208Z Entering 'third_party/XNNPACK'
2023-01-11T20:43:38.1112168Z Entering 'third_party/benchmark'
2023-01-11T20:43:38.1147065Z Entering 'third_party/cpuinfo'
2023-01-11T20:43:38.1181638Z Entering 'third_party/cub'
2023-01-11T20:43:38.1216306Z Entering 'third_party/cudnn_frontend'
2023-01-11T20:43:38.1256128Z Entering 'third_party/cutlass'
2023-01-11T20:43:38.1296434Z Entering 'third_party/eigen'
2023-01-11T20:43:38.1332340Z Entering 'third_party/fbgemm'
2023-01-11T20:43:38.1366989Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T20:43:38.1402213Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T20:43:38.1436056Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T20:43:38.1469503Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T20:43:38.1503643Z Entering 'third_party/flatbuffers'
2023-01-11T20:43:38.1578031Z Entering 'third_party/fmt'
2023-01-11T20:43:38.1579381Z Entering 'third_party/foxi'
2023-01-11T20:43:38.1614451Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T20:43:38.1649108Z Entering 'third_party/gloo'
2023-01-11T20:43:38.1684727Z Entering 'third_party/googletest'
2023-01-11T20:43:38.1719052Z Entering 'third_party/ideep'
2023-01-11T20:43:38.1753458Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T20:43:38.1789482Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T20:43:38.1829864Z Entering 'third_party/ios-cmake'
2023-01-11T20:43:38.1865595Z Entering 'third_party/ittapi'
2023-01-11T20:43:38.1900729Z Entering 'third_party/kineto'
2023-01-11T20:43:38.1935144Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T20:43:38.1968084Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T20:43:38.2003185Z Entering 'third_party/nccl/nccl'
2023-01-11T20:43:38.2037589Z Entering 'third_party/neon2sse'
2023-01-11T20:43:38.2072460Z Entering 'third_party/nlohmann'
2023-01-11T20:43:38.2108454Z Entering 'third_party/onnx'
2023-01-11T20:43:38.2154410Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T20:43:38.2188303Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T20:43:38.2224636Z Entering 'third_party/onnx-tensorrt'
2023-01-11T20:43:38.2258813Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T20:43:38.2296500Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T20:43:38.2329950Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T20:43:38.2363507Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T20:43:38.2401354Z Entering 'third_party/pocketfft'
2023-01-11T20:43:38.2435342Z Entering 'third_party/protobuf'
2023-01-11T20:43:38.2472679Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T20:43:38.2506204Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T20:43:38.2541663Z Entering 'third_party/psimd'
2023-01-11T20:43:38.2575753Z Entering 'third_party/pthreadpool'
2023-01-11T20:43:38.2609345Z Entering 'third_party/pybind11'
2023-01-11T20:43:38.2643160Z Entering 'third_party/python-enum'
2023-01-11T20:43:38.2676769Z Entering 'third_party/python-peachpy'
2023-01-11T20:43:38.2710972Z Entering 'third_party/python-six'
2023-01-11T20:43:38.2744400Z Entering 'third_party/sleef'
2023-01-11T20:43:38.2778551Z Entering 'third_party/tbb'
2023-01-11T20:43:38.2815261Z Entering 'third_party/tensorpipe'
2023-01-11T20:43:38.2849064Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T20:43:38.2882529Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T20:43:38.2916576Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T20:43:38.2951450Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T20:43:38.2984409Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T20:43:38.3020927Z Entering 'third_party/zstd'
2023-01-11T20:43:38.3062655Z ##[endgroup]
2023-01-11T20:43:38.3063022Z ##[group]Persisting credentials for submodules
2023-01-11T20:43:38.3068860Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2023-01-11T20:43:38.3320037Z Entering 'android/libs/fbjni'
2023-01-11T20:43:38.3354426Z Entering 'third_party/FP16'
2023-01-11T20:43:38.3388275Z Entering 'third_party/FXdiv'
2023-01-11T20:43:38.3421459Z Entering 'third_party/NNPACK'
2023-01-11T20:43:38.3454620Z Entering 'third_party/QNNPACK'
2023-01-11T20:43:38.3488296Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T20:43:38.3523059Z Entering 'third_party/XNNPACK'
2023-01-11T20:43:38.3566206Z Entering 'third_party/benchmark'
2023-01-11T20:43:38.3599771Z Entering 'third_party/cpuinfo'
2023-01-11T20:43:38.3633132Z Entering 'third_party/cub'
2023-01-11T20:43:38.3666943Z Entering 'third_party/cudnn_frontend'
2023-01-11T20:43:38.3705078Z Entering 'third_party/cutlass'
2023-01-11T20:43:38.3743647Z Entering 'third_party/eigen'
2023-01-11T20:43:38.3778980Z Entering 'third_party/fbgemm'
2023-01-11T20:43:38.3812188Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T20:43:38.3845070Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T20:43:38.3878178Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T20:43:38.3910611Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T20:43:38.3945426Z Entering 'third_party/flatbuffers'
2023-01-11T20:43:38.3980549Z Entering 'third_party/fmt'
2023-01-11T20:43:38.4014437Z Entering 'third_party/foxi'
2023-01-11T20:43:38.4047040Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T20:43:38.4080796Z Entering 'third_party/gloo'
2023-01-11T20:43:38.4114213Z Entering 'third_party/googletest'
2023-01-11T20:43:38.4147890Z Entering 'third_party/ideep'
2023-01-11T20:43:38.4180676Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T20:43:38.4215992Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T20:43:38.4254898Z Entering 'third_party/ios-cmake'
2023-01-11T20:43:38.4288080Z Entering 'third_party/ittapi'
2023-01-11T20:43:38.4322427Z Entering 'third_party/kineto'
2023-01-11T20:43:38.4355432Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T20:43:38.4389295Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T20:43:38.4423629Z Entering 'third_party/nccl/nccl'
2023-01-11T20:43:38.4457034Z Entering 'third_party/neon2sse'
2023-01-11T20:43:38.4490426Z Entering 'third_party/nlohmann'
2023-01-11T20:43:38.4524536Z Entering 'third_party/onnx'
2023-01-11T20:43:38.4569174Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T20:43:38.4602428Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T20:43:38.4637734Z Entering 'third_party/onnx-tensorrt'
2023-01-11T20:43:38.4672204Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T20:43:38.4707953Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T20:43:38.4741294Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T20:43:38.4775365Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T20:43:38.4811870Z Entering 'third_party/pocketfft'
2023-01-11T20:43:38.4844874Z Entering 'third_party/protobuf'
2023-01-11T20:43:38.4881465Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T20:43:38.4915093Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T20:43:38.4951860Z Entering 'third_party/psimd'
2023-01-11T20:43:38.4984459Z Entering 'third_party/pthreadpool'
2023-01-11T20:43:38.5018499Z Entering 'third_party/pybind11'
2023-01-11T20:43:38.5052334Z Entering 'third_party/python-enum'
2023-01-11T20:43:38.5084572Z Entering 'third_party/python-peachpy'
2023-01-11T20:43:38.5118729Z Entering 'third_party/python-six'
2023-01-11T20:43:38.5151864Z Entering 'third_party/sleef'
2023-01-11T20:43:38.5185623Z Entering 'third_party/tbb'
2023-01-11T20:43:38.5220911Z Entering 'third_party/tensorpipe'
2023-01-11T20:43:38.5256040Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T20:43:38.5288628Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T20:43:38.5321049Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T20:43:38.5354619Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T20:43:38.5388570Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T20:43:38.5423673Z Entering 'third_party/zstd'
2023-01-11T20:43:38.5468181Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2023-01-11T20:43:38.5720153Z Entering 'android/libs/fbjni'
2023-01-11T20:43:38.5750980Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2023-01-11T20:43:38.5765207Z Entering 'third_party/FP16'
2023-01-11T20:43:38.5797788Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2023-01-11T20:43:38.5812077Z Entering 'third_party/FXdiv'
2023-01-11T20:43:38.5843248Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2023-01-11T20:43:38.5857671Z Entering 'third_party/NNPACK'
2023-01-11T20:43:38.5888765Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2023-01-11T20:43:38.5903408Z Entering 'third_party/QNNPACK'
2023-01-11T20:43:38.5934292Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2023-01-11T20:43:38.5948877Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T20:43:38.5981109Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2023-01-11T20:43:38.5995282Z Entering 'third_party/XNNPACK'
2023-01-11T20:43:38.6027935Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2023-01-11T20:43:38.6050653Z Entering 'third_party/benchmark'
2023-01-11T20:43:38.6082571Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T20:43:38.6098329Z Entering 'third_party/cpuinfo'
2023-01-11T20:43:38.6128894Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2023-01-11T20:43:38.6143695Z Entering 'third_party/cub'
2023-01-11T20:43:38.6175827Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2023-01-11T20:43:38.6189403Z Entering 'third_party/cudnn_frontend'
2023-01-11T20:43:38.6221954Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2023-01-11T20:43:38.6241099Z Entering 'third_party/cutlass'
2023-01-11T20:43:38.6274135Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2023-01-11T20:43:38.6294804Z Entering 'third_party/eigen'
2023-01-11T20:43:38.6326199Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2023-01-11T20:43:38.6342964Z Entering 'third_party/fbgemm'
2023-01-11T20:43:38.6374689Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2023-01-11T20:43:38.6388391Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T20:43:38.6420290Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2023-01-11T20:43:38.6433916Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T20:43:38.6464600Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2023-01-11T20:43:38.6478396Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T20:43:38.6510206Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2023-01-11T20:43:38.6524360Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T20:43:38.6556420Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2023-01-11T20:43:38.6571327Z Entering 'third_party/flatbuffers'
2023-01-11T20:43:38.6603307Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2023-01-11T20:43:38.6619362Z Entering 'third_party/fmt'
2023-01-11T20:43:38.6650991Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2023-01-11T20:43:38.6665852Z Entering 'third_party/foxi'
2023-01-11T20:43:38.6698030Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2023-01-11T20:43:38.6711405Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T20:43:38.6743698Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2023-01-11T20:43:38.6758258Z Entering 'third_party/gloo'
2023-01-11T20:43:38.6789355Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2023-01-11T20:43:38.6803761Z Entering 'third_party/googletest'
2023-01-11T20:43:38.6835213Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2023-01-11T20:43:38.6849684Z Entering 'third_party/ideep'
2023-01-11T20:43:38.6881389Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2023-01-11T20:43:38.6895542Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T20:43:38.6926686Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2023-01-11T20:43:38.6942744Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T20:43:38.6974137Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2023-01-11T20:43:38.6994473Z Entering 'third_party/ios-cmake'
2023-01-11T20:43:38.7027306Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2023-01-11T20:43:38.7040864Z Entering 'third_party/ittapi'
2023-01-11T20:43:38.7072267Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2023-01-11T20:43:38.7085767Z Entering 'third_party/kineto'
2023-01-11T20:43:38.7117353Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2023-01-11T20:43:38.7131466Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T20:43:38.7163206Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2023-01-11T20:43:38.7178058Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T20:43:38.7209610Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2023-01-11T20:43:38.7224760Z Entering 'third_party/nccl/nccl'
2023-01-11T20:43:38.7257504Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2023-01-11T20:43:38.7271338Z Entering 'third_party/neon2sse'
2023-01-11T20:43:38.7303890Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2023-01-11T20:43:38.7317394Z Entering 'third_party/nlohmann'
2023-01-11T20:43:38.7349354Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2023-01-11T20:43:38.7364010Z Entering 'third_party/onnx'
2023-01-11T20:43:38.7396265Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2023-01-11T20:43:38.7421725Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T20:43:38.7453378Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T20:43:38.7467829Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T20:43:38.7500136Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T20:43:38.7516429Z Entering 'third_party/onnx-tensorrt'
2023-01-11T20:43:38.7548477Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2023-01-11T20:43:38.7561652Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T20:43:38.7593913Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2023-01-11T20:43:38.7613062Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T20:43:38.7644345Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T20:43:38.7658661Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T20:43:38.7689979Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T20:43:38.7703596Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T20:43:38.7735739Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2023-01-11T20:43:38.7753956Z Entering 'third_party/pocketfft'
2023-01-11T20:43:38.7786426Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2023-01-11T20:43:38.7800551Z Entering 'third_party/protobuf'
2023-01-11T20:43:38.7832595Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2023-01-11T20:43:38.7850231Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T20:43:38.7881799Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T20:43:38.7895491Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T20:43:38.7926515Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2023-01-11T20:43:38.7942400Z Entering 'third_party/psimd'
2023-01-11T20:43:38.7975144Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2023-01-11T20:43:38.7989032Z Entering 'third_party/pthreadpool'
2023-01-11T20:43:38.8022320Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2023-01-11T20:43:38.8036326Z Entering 'third_party/pybind11'
2023-01-11T20:43:38.8068208Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T20:43:38.8082875Z Entering 'third_party/python-enum'
2023-01-11T20:43:38.8114715Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2023-01-11T20:43:38.8128536Z Entering 'third_party/python-peachpy'
2023-01-11T20:43:38.8159761Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2023-01-11T20:43:38.8174422Z Entering 'third_party/python-six'
2023-01-11T20:43:38.8206016Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2023-01-11T20:43:38.8220434Z Entering 'third_party/sleef'
2023-01-11T20:43:38.8251643Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2023-01-11T20:43:38.8265695Z Entering 'third_party/tbb'
2023-01-11T20:43:38.8297377Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2023-01-11T20:43:38.8313697Z Entering 'third_party/tensorpipe'
2023-01-11T20:43:38.8346199Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2023-01-11T20:43:38.8360234Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T20:43:38.8392767Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2023-01-11T20:43:38.8407663Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T20:43:38.8439423Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2023-01-11T20:43:38.8453900Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T20:43:38.8485225Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2023-01-11T20:43:38.8499826Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T20:43:38.8532464Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T20:43:38.8545256Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T20:43:38.8577040Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2023-01-11T20:43:38.8594104Z Entering 'third_party/zstd'
2023-01-11T20:43:38.8625867Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2023-01-11T20:43:38.9265229Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2023-01-11T20:43:38.9527607Z Entering 'android/libs/fbjni'
2023-01-11T20:43:38.9562530Z Entering 'third_party/FP16'
2023-01-11T20:43:38.9598886Z Entering 'third_party/FXdiv'
2023-01-11T20:43:38.9633431Z Entering 'third_party/NNPACK'
2023-01-11T20:43:38.9668305Z Entering 'third_party/QNNPACK'
2023-01-11T20:43:38.9703455Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T20:43:38.9738277Z Entering 'third_party/XNNPACK'
2023-01-11T20:43:38.9782334Z Entering 'third_party/benchmark'
2023-01-11T20:43:38.9816970Z Entering 'third_party/cpuinfo'
2023-01-11T20:43:38.9852047Z Entering 'third_party/cub'
2023-01-11T20:43:38.9886393Z Entering 'third_party/cudnn_frontend'
2023-01-11T20:43:38.9926273Z Entering 'third_party/cutlass'
2023-01-11T20:43:38.9967237Z Entering 'third_party/eigen'
2023-01-11T20:43:39.0004257Z Entering 'third_party/fbgemm'
2023-01-11T20:43:39.0039094Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T20:43:39.0072623Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T20:43:39.0108490Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T20:43:39.0141932Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T20:43:39.0177550Z Entering 'third_party/flatbuffers'
2023-01-11T20:43:39.0213898Z Entering 'third_party/fmt'
2023-01-11T20:43:39.0248024Z Entering 'third_party/foxi'
2023-01-11T20:43:39.0282080Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T20:43:39.0317373Z Entering 'third_party/gloo'
2023-01-11T20:43:39.0353600Z Entering 'third_party/googletest'
2023-01-11T20:43:39.0388627Z Entering 'third_party/ideep'
2023-01-11T20:43:39.0422952Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T20:43:39.0459320Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T20:43:39.0500135Z Entering 'third_party/ios-cmake'
2023-01-11T20:43:39.0535551Z Entering 'third_party/ittapi'
2023-01-11T20:43:39.0569414Z Entering 'third_party/kineto'
2023-01-11T20:43:39.0604306Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T20:43:39.0637905Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T20:43:39.0672695Z Entering 'third_party/nccl/nccl'
2023-01-11T20:43:39.0707534Z Entering 'third_party/neon2sse'
2023-01-11T20:43:39.0741214Z Entering 'third_party/nlohmann'
2023-01-11T20:43:39.0776589Z Entering 'third_party/onnx'
2023-01-11T20:43:39.0821947Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T20:43:39.0857044Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T20:43:39.0893142Z Entering 'third_party/onnx-tensorrt'
2023-01-11T20:43:39.0927467Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T20:43:39.0965004Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T20:43:39.1000498Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T20:43:39.1034385Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T20:43:39.1072629Z Entering 'third_party/pocketfft'
2023-01-11T20:43:39.1107027Z Entering 'third_party/protobuf'
2023-01-11T20:43:39.1144069Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T20:43:39.1178064Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T20:43:39.1214552Z Entering 'third_party/psimd'
2023-01-11T20:43:39.1248134Z Entering 'third_party/pthreadpool'
2023-01-11T20:43:39.1282361Z Entering 'third_party/pybind11'
2023-01-11T20:43:39.1316674Z Entering 'third_party/python-enum'
2023-01-11T20:43:39.1352519Z Entering 'third_party/python-peachpy'
2023-01-11T20:43:39.1386816Z Entering 'third_party/python-six'
2023-01-11T20:43:39.1420999Z Entering 'third_party/sleef'
2023-01-11T20:43:39.1455137Z Entering 'third_party/tbb'
2023-01-11T20:43:39.1492316Z Entering 'third_party/tensorpipe'
2023-01-11T20:43:39.1526021Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T20:43:39.1559977Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T20:43:39.1594003Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T20:43:39.1628128Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T20:43:39.1661914Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T20:43:39.1698804Z Entering 'third_party/zstd'
2023-01-11T20:43:39.1744331Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2023-01-11T20:43:39.1996018Z Entering 'android/libs/fbjni'
2023-01-11T20:43:39.2029422Z Entering 'third_party/FP16'
2023-01-11T20:43:39.2065302Z Entering 'third_party/FXdiv'
2023-01-11T20:43:39.2100155Z Entering 'third_party/NNPACK'
2023-01-11T20:43:39.2136420Z Entering 'third_party/QNNPACK'
2023-01-11T20:43:39.2170235Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T20:43:39.2205140Z Entering 'third_party/XNNPACK'
2023-01-11T20:43:39.2249137Z Entering 'third_party/benchmark'
2023-01-11T20:43:39.2283757Z Entering 'third_party/cpuinfo'
2023-01-11T20:43:39.2318310Z Entering 'third_party/cub'
2023-01-11T20:43:39.2352846Z Entering 'third_party/cudnn_frontend'
2023-01-11T20:43:39.2392511Z Entering 'third_party/cutlass'
2023-01-11T20:43:39.2432877Z Entering 'third_party/eigen'
2023-01-11T20:43:39.2469998Z Entering 'third_party/fbgemm'
2023-01-11T20:43:39.2504219Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T20:43:39.2539250Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T20:43:39.2573259Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T20:43:39.2606530Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T20:43:39.2640906Z Entering 'third_party/flatbuffers'
2023-01-11T20:43:39.2678501Z Entering 'third_party/fmt'
2023-01-11T20:43:39.2712894Z Entering 'third_party/foxi'
2023-01-11T20:43:39.2747931Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T20:43:39.2782632Z Entering 'third_party/gloo'
2023-01-11T20:43:39.2817204Z Entering 'third_party/googletest'
2023-01-11T20:43:39.2851381Z Entering 'third_party/ideep'
2023-01-11T20:43:39.2884915Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T20:43:39.2920952Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T20:43:39.2961768Z Entering 'third_party/ios-cmake'
2023-01-11T20:43:39.2995374Z Entering 'third_party/ittapi'
2023-01-11T20:43:39.3030081Z Entering 'third_party/kineto'
2023-01-11T20:43:39.3065774Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T20:43:39.3099604Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T20:43:39.3134642Z Entering 'third_party/nccl/nccl'
2023-01-11T20:43:39.3169613Z Entering 'third_party/neon2sse'
2023-01-11T20:43:39.3203439Z Entering 'third_party/nlohmann'
2023-01-11T20:43:39.3238789Z Entering 'third_party/onnx'
2023-01-11T20:43:39.3284167Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T20:43:39.3317807Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T20:43:39.3353838Z Entering 'third_party/onnx-tensorrt'
2023-01-11T20:43:39.3388399Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T20:43:39.3426433Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T20:43:39.3460835Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T20:43:39.3494166Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T20:43:39.3531159Z Entering 'third_party/pocketfft'
2023-01-11T20:43:39.3564541Z Entering 'third_party/protobuf'
2023-01-11T20:43:39.3601870Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T20:43:39.3636371Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T20:43:39.3671794Z Entering 'third_party/psimd'
2023-01-11T20:43:39.3707045Z Entering 'third_party/pthreadpool'
2023-01-11T20:43:39.3740933Z Entering 'third_party/pybind11'
2023-01-11T20:43:39.3774772Z Entering 'third_party/python-enum'
2023-01-11T20:43:39.3808855Z Entering 'third_party/python-peachpy'
2023-01-11T20:43:39.3843300Z Entering 'third_party/python-six'
2023-01-11T20:43:39.3878659Z Entering 'third_party/sleef'
2023-01-11T20:43:39.3913106Z Entering 'third_party/tbb'
2023-01-11T20:43:39.3949521Z Entering 'third_party/tensorpipe'
2023-01-11T20:43:39.3985169Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T20:43:39.4018958Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T20:43:39.4052416Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T20:43:39.4084922Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T20:43:39.4117914Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T20:43:39.4154450Z Entering 'third_party/zstd'
2023-01-11T20:43:39.4195075Z ##[endgroup]
2023-01-11T20:43:39.4232775Z [command]/usr/bin/git log -1 --format='%H'
2023-01-11T20:43:39.4258302Z '8419ddda87c8a47eacc63b54bc7ec98c1f27c26e'
2023-01-11T20:43:39.4374465Z Prepare all required actions
2023-01-11T20:43:39.4399760Z ##[group]Run ./.github/actions/setup-linux
2023-01-11T20:43:39.4399946Z env:
2023-01-11T20:43:39.4400122Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:43:39.4400313Z ##[endgroup]
2023-01-11T20:43:39.4415316Z ##[group]Run set -euo pipefail
2023-01-11T20:43:39.4415550Z set -euo pipefail
2023-01-11T20:43:39.4415761Z function get_ec2_metadata() {
2023-01-11T20:43:39.4415991Z   # Pulled from instance metadata endpoint for EC2
2023-01-11T20:43:39.4416345Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2023-01-11T20:43:39.4416652Z   category=$1
2023-01-11T20:43:39.4416897Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2023-01-11T20:43:39.4417108Z }
2023-01-11T20:43:39.4417309Z echo "ami-id: $(get_ec2_metadata ami-id)"
2023-01-11T20:43:39.4417593Z echo "instance-id: $(get_ec2_metadata instance-id)"
2023-01-11T20:43:39.4417853Z echo "instance-type: $(get_ec2_metadata instance-type)"
2023-01-11T20:43:39.4418100Z echo "system info $(uname -a)"
2023-01-11T20:43:39.4429063Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T20:43:39.4429268Z env:
2023-01-11T20:43:39.4429444Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:43:39.4429634Z ##[endgroup]
2023-01-11T20:43:39.4507533Z ami-id: ami-096198a0bccc6bad4
2023-01-11T20:43:39.4557409Z instance-id: i-01f7d826cf45480d0
2023-01-11T20:43:39.4605664Z instance-type: c5.2xlarge
2023-01-11T20:43:39.4612003Z system info Linux ip-10-0-1-8.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2023-01-11T20:43:39.4627039Z ##[group]Run if systemctl is-active --quiet docker; then
2023-01-11T20:43:39.4627319Z if systemctl is-active --quiet docker; then
2023-01-11T20:43:39.4627549Z     echo "Docker daemon is running...";
2023-01-11T20:43:39.4627756Z else
2023-01-11T20:43:39.4627987Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2023-01-11T20:43:39.4628213Z fi
2023-01-11T20:43:39.4638502Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T20:43:39.4638716Z env:
2023-01-11T20:43:39.4638889Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:43:39.4639064Z ##[endgroup]
2023-01-11T20:43:39.4677067Z Docker daemon is running...
2023-01-11T20:43:39.4692512Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2023-01-11T20:43:39.4692863Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2023-01-11T20:43:39.4693152Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2023-01-11T20:43:39.4693522Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2023-01-11T20:43:39.4693857Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2023-01-11T20:43:39.4704135Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T20:43:39.4704356Z env:
2023-01-11T20:43:39.4704517Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:43:39.4704711Z   AWS_RETRY_MODE: standard
2023-01-11T20:43:39.4704898Z   AWS_MAX_ATTEMPTS: 5
2023-01-11T20:43:39.4705080Z   AWS_DEFAULT_REGION: us-east-1
2023-01-11T20:43:39.4705268Z ##[endgroup]
2023-01-11T20:43:40.2126050Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2023-01-11T20:43:40.2126399Z Configure a credential helper to remove this warning. See
2023-01-11T20:43:40.2127056Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2023-01-11T20:43:40.2127331Z 
2023-01-11T20:43:40.2127661Z Login Succeeded
2023-01-11T20:43:40.2157073Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-01-11T20:43:40.2157387Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-01-11T20:43:40.2157734Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-01-11T20:43:40.2168638Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T20:43:40.2168843Z env:
2023-01-11T20:43:40.2169018Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:43:40.2169205Z ##[endgroup]
2023-01-11T20:43:40.2250739Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2023-01-11T20:43:40.2251418Z with:
2023-01-11T20:43:40.2252021Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T20:43:40.2252636Z env:
2023-01-11T20:43:40.2252968Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:43:40.2253291Z ##[endgroup]
2023-01-11T20:43:40.2269791Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2023-01-11T20:43:40.2270090Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2023-01-11T20:43:40.2270382Z # ignore output since only exit code is used for conditional
2023-01-11T20:43:40.2270691Z # only pull docker image if it's not available locally
2023-01-11T20:43:40.2271013Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2023-01-11T20:43:40.2271338Z   retry docker pull "${DOCKER_IMAGE}"
2023-01-11T20:43:40.2271563Z fi
2023-01-11T20:43:40.2286430Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T20:43:40.2286865Z env:
2023-01-11T20:43:40.2287238Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:43:40.2287899Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T20:43:40.2288527Z ##[endgroup]
2023-01-11T20:43:40.5029872Z fd224c2e6c79d7fdec6408da598bf52bc5b201dd: Pulling from pytorch/pytorch-linux-focal-py3.7-gcc7
2023-01-11T20:43:40.5030801Z 846c0b181fff: Pulling fs layer
2023-01-11T20:43:40.5031232Z 0698304963f1: Pulling fs layer
2023-01-11T20:43:40.5034240Z 77ce1d4f36bf: Pulling fs layer
2023-01-11T20:43:40.5034666Z d39938d48fa2: Pulling fs layer
2023-01-11T20:43:40.5034989Z ae4dac5a8919: Pulling fs layer
2023-01-11T20:43:40.5035328Z 4ff68f5f773b: Pulling fs layer
2023-01-11T20:43:40.5035602Z 47642f12aa0f: Pulling fs layer
2023-01-11T20:43:40.5035854Z 3c02291730dc: Pulling fs layer
2023-01-11T20:43:40.5036048Z c7165c4f5e23: Pulling fs layer
2023-01-11T20:43:40.5036243Z 8357faec0a0e: Pulling fs layer
2023-01-11T20:43:40.5036441Z 42fee15d0ad3: Pulling fs layer
2023-01-11T20:43:40.5036634Z 246016b9e8c8: Pulling fs layer
2023-01-11T20:43:40.5036815Z 25be21861eed: Pulling fs layer
2023-01-11T20:43:40.5037013Z 21ac276b84eb: Pulling fs layer
2023-01-11T20:43:40.5037197Z d39938d48fa2: Waiting
2023-01-11T20:43:40.5037370Z bd695a285c41: Pulling fs layer
2023-01-11T20:43:40.5040240Z 9af030ba5896: Pulling fs layer
2023-01-11T20:43:40.5040759Z 3c02291730dc: Waiting
2023-01-11T20:43:40.5041097Z ae4dac5a8919: Waiting
2023-01-11T20:43:40.5041412Z 0ab7fbe28a03: Pulling fs layer
2023-01-11T20:43:40.5041713Z f5dba516d75a: Pulling fs layer
2023-01-11T20:43:40.5041961Z 944388bd6de3: Pulling fs layer
2023-01-11T20:43:40.5042277Z 32233288f470: Pulling fs layer
2023-01-11T20:43:40.5042590Z 4ff68f5f773b: Waiting
2023-01-11T20:43:40.5042920Z 4716e681e05e: Pulling fs layer
2023-01-11T20:43:40.5043117Z 47642f12aa0f: Waiting
2023-01-11T20:43:40.5043285Z 07008cd698a0: Pulling fs layer
2023-01-11T20:43:40.5043474Z 27f046e694b1: Pulling fs layer
2023-01-11T20:43:40.5043657Z 8357faec0a0e: Waiting
2023-01-11T20:43:40.5043826Z 95f903a2e692: Pulling fs layer
2023-01-11T20:43:40.5044005Z 42fee15d0ad3: Waiting
2023-01-11T20:43:40.5053321Z 1d465abcef1e: Pulling fs layer
2023-01-11T20:43:40.5053826Z 32fdce95f2af: Pulling fs layer
2023-01-11T20:43:40.5054151Z 38f17f1788cc: Pulling fs layer
2023-01-11T20:43:40.5054449Z bd695a285c41: Waiting
2023-01-11T20:43:40.5054702Z fa2049e24ad0: Pulling fs layer
2023-01-11T20:43:40.5054898Z 070d94af58e3: Pulling fs layer
2023-01-11T20:43:40.5055164Z 9af030ba5896: Waiting
2023-01-11T20:43:40.5055342Z 9ff218827ef0: Pulling fs layer
2023-01-11T20:43:40.5055525Z 0ab7fbe28a03: Waiting
2023-01-11T20:43:40.5055711Z 0effc56e2dd9: Pulling fs layer
2023-01-11T20:43:40.5055885Z f5dba516d75a: Waiting
2023-01-11T20:43:40.5056068Z bd453ee44627: Pulling fs layer
2023-01-11T20:43:40.5056410Z e999c29eb8b9: Pulling fs layer
2023-01-11T20:43:40.5056595Z 566e975723c3: Pulling fs layer
2023-01-11T20:43:40.5056809Z 1072a14c5615: Pulling fs layer
2023-01-11T20:43:40.5057136Z 944388bd6de3: Waiting
2023-01-11T20:43:40.5057420Z 9ae70c90b543: Pulling fs layer
2023-01-11T20:43:40.5057691Z 32233288f470: Waiting
2023-01-11T20:43:40.5057910Z 38f17f1788cc: Waiting
2023-01-11T20:43:40.5058077Z e6f8bf185266: Pulling fs layer
2023-01-11T20:43:40.5058259Z fa2049e24ad0: Waiting
2023-01-11T20:43:40.5058438Z 01ff2898de6c: Pulling fs layer
2023-01-11T20:43:40.5058702Z 98f07e351fe7: Pulling fs layer
2023-01-11T20:43:40.5058975Z 070d94af58e3: Waiting
2023-01-11T20:43:40.5059220Z 07008cd698a0: Waiting
2023-01-11T20:43:40.5059401Z e426b18803bb: Pulling fs layer
2023-01-11T20:43:40.5059568Z 9ff218827ef0: Waiting
2023-01-11T20:43:40.5059908Z da3dd65b2531: Pulling fs layer
2023-01-11T20:43:40.5060412Z 27f046e694b1: Waiting
2023-01-11T20:43:40.5060705Z 5c3ec8735bee: Pulling fs layer
2023-01-11T20:43:40.5060994Z b616c89406f0: Pulling fs layer
2023-01-11T20:43:40.5061293Z f48f88ff2185: Pulling fs layer
2023-01-11T20:43:40.5061504Z 1072a14c5615: Waiting
2023-01-11T20:43:40.5061779Z 0effc56e2dd9: Waiting
2023-01-11T20:43:40.5062074Z 46e0cb4c14f2: Pulling fs layer
2023-01-11T20:43:40.5062377Z bd453ee44627: Waiting
2023-01-11T20:43:40.5062716Z 300515e4431f: Pulling fs layer
2023-01-11T20:43:40.5063030Z 9df79d1a6e27: Pulling fs layer
2023-01-11T20:43:40.5063364Z 3cda6364e15e: Pulling fs layer
2023-01-11T20:43:40.5063706Z 9ae70c90b543: Waiting
2023-01-11T20:43:40.5064027Z e6f8bf185266: Waiting
2023-01-11T20:43:40.5064226Z 21ac276b84eb: Waiting
2023-01-11T20:43:40.5064409Z 7fc4121e1952: Pulling fs layer
2023-01-11T20:43:40.5064622Z 676dcfb6d5c2: Pulling fs layer
2023-01-11T20:43:40.5064805Z 98f07e351fe7: Waiting
2023-01-11T20:43:40.5064963Z 25be21861eed: Waiting
2023-01-11T20:43:40.5065143Z 130a71e99e46: Pulling fs layer
2023-01-11T20:43:40.5065321Z e426b18803bb: Waiting
2023-01-11T20:43:40.5065486Z 932461d264d4: Pulling fs layer
2023-01-11T20:43:40.5065677Z 6cce994e24e8: Pulling fs layer
2023-01-11T20:43:40.5065866Z b37b5d679247: Pulling fs layer
2023-01-11T20:43:40.5066033Z da3dd65b2531: Waiting
2023-01-11T20:43:40.5066216Z 86ad3a500f17: Pulling fs layer
2023-01-11T20:43:40.5066394Z 95f903a2e692: Waiting
2023-01-11T20:43:40.5066562Z 93e8483d7d76: Pulling fs layer
2023-01-11T20:43:40.5066755Z ad1ecb94a573: Pulling fs layer
2023-01-11T20:43:40.5066940Z 5c3ec8735bee: Waiting
2023-01-11T20:43:40.5067101Z e999c29eb8b9: Waiting
2023-01-11T20:43:40.5067286Z a7f9c3a7c21d: Pulling fs layer
2023-01-11T20:43:40.5067467Z f48f88ff2185: Waiting
2023-01-11T20:43:40.5067630Z 46e0cb4c14f2: Waiting
2023-01-11T20:43:40.5067796Z 300515e4431f: Waiting
2023-01-11T20:43:40.5067960Z 932461d264d4: Waiting
2023-01-11T20:43:40.5068297Z 32fdce95f2af: Waiting
2023-01-11T20:43:40.5068471Z 9df79d1a6e27: Waiting
2023-01-11T20:43:40.5068643Z 1d465abcef1e: Waiting
2023-01-11T20:43:40.5068800Z 6cce994e24e8: Waiting
2023-01-11T20:43:40.5068970Z 3cda6364e15e: Waiting
2023-01-11T20:43:40.5069140Z 566e975723c3: Waiting
2023-01-11T20:43:40.5069292Z 93e8483d7d76: Waiting
2023-01-11T20:43:40.5069461Z 86ad3a500f17: Waiting
2023-01-11T20:43:40.5069630Z a7f9c3a7c21d: Waiting
2023-01-11T20:43:40.5069785Z ad1ecb94a573: Waiting
2023-01-11T20:43:40.5070049Z 130a71e99e46: Waiting
2023-01-11T20:43:40.5070214Z b616c89406f0: Waiting
2023-01-11T20:43:40.5070485Z 7fc4121e1952: Waiting
2023-01-11T20:43:40.6503080Z 0698304963f1: Verifying Checksum
2023-01-11T20:43:40.6503647Z 0698304963f1: Download complete
2023-01-11T20:43:40.7209073Z d39938d48fa2: Verifying Checksum
2023-01-11T20:43:40.7209539Z d39938d48fa2: Download complete
2023-01-11T20:43:40.8006199Z ae4dac5a8919: Verifying Checksum
2023-01-11T20:43:40.8006502Z ae4dac5a8919: Download complete
2023-01-11T20:43:40.8485267Z 846c0b181fff: Verifying Checksum
2023-01-11T20:43:40.8485672Z 846c0b181fff: Download complete
2023-01-11T20:43:40.8945696Z 4ff68f5f773b: Verifying Checksum
2023-01-11T20:43:40.8946107Z 4ff68f5f773b: Download complete
2023-01-11T20:43:40.9388907Z 47642f12aa0f: Verifying Checksum
2023-01-11T20:43:40.9389386Z 47642f12aa0f: Download complete
2023-01-11T20:43:40.9831685Z 3c02291730dc: Verifying Checksum
2023-01-11T20:43:40.9832158Z 3c02291730dc: Download complete
2023-01-11T20:43:41.0225280Z c7165c4f5e23: Verifying Checksum
2023-01-11T20:43:41.0225561Z c7165c4f5e23: Download complete
2023-01-11T20:43:41.0597266Z 8357faec0a0e: Verifying Checksum
2023-01-11T20:43:41.0597567Z 8357faec0a0e: Download complete
2023-01-11T20:43:41.1545740Z 246016b9e8c8: Verifying Checksum
2023-01-11T20:43:41.1546147Z 246016b9e8c8: Download complete
2023-01-11T20:43:41.2216818Z 25be21861eed: Download complete
2023-01-11T20:43:41.5342893Z 846c0b181fff: Pull complete
2023-01-11T20:43:41.6499882Z 0698304963f1: Pull complete
2023-01-11T20:43:41.9002409Z 42fee15d0ad3: Verifying Checksum
2023-01-11T20:43:41.9002705Z 42fee15d0ad3: Download complete
2023-01-11T20:43:41.9857154Z bd695a285c41: Download complete
2023-01-11T20:43:42.0621608Z 9af030ba5896: Verifying Checksum
2023-01-11T20:43:42.0621946Z 9af030ba5896: Download complete
2023-01-11T20:43:42.5697178Z 0ab7fbe28a03: Verifying Checksum
2023-01-11T20:43:42.5697499Z 0ab7fbe28a03: Download complete
2023-01-11T20:43:42.6547280Z f5dba516d75a: Download complete
2023-01-11T20:43:42.7763775Z 944388bd6de3: Verifying Checksum
2023-01-11T20:43:42.7764144Z 944388bd6de3: Download complete
2023-01-11T20:43:42.8650257Z 32233288f470: Download complete
2023-01-11T20:43:42.9365101Z 4716e681e05e: Download complete
2023-01-11T20:43:43.0100474Z 07008cd698a0: Verifying Checksum
2023-01-11T20:43:43.0100729Z 07008cd698a0: Download complete
2023-01-11T20:43:43.0901373Z 27f046e694b1: Verifying Checksum
2023-01-11T20:43:43.0901878Z 27f046e694b1: Download complete
2023-01-11T20:43:43.1691955Z 95f903a2e692: Verifying Checksum
2023-01-11T20:43:43.1692307Z 95f903a2e692: Download complete
2023-01-11T20:43:43.7760740Z 77ce1d4f36bf: Verifying Checksum
2023-01-11T20:43:43.7761263Z 77ce1d4f36bf: Download complete
2023-01-11T20:43:43.8539176Z 32fdce95f2af: Download complete
2023-01-11T20:43:43.9565639Z 38f17f1788cc: Verifying Checksum
2023-01-11T20:43:43.9566037Z 38f17f1788cc: Download complete
2023-01-11T20:43:44.0520087Z fa2049e24ad0: Verifying Checksum
2023-01-11T20:43:44.0520524Z fa2049e24ad0: Download complete
2023-01-11T20:43:44.1060923Z 1d465abcef1e: Verifying Checksum
2023-01-11T20:43:44.1061446Z 1d465abcef1e: Download complete
2023-01-11T20:43:44.1482930Z 070d94af58e3: Verifying Checksum
2023-01-11T20:43:44.1483363Z 070d94af58e3: Download complete
2023-01-11T20:43:44.1874835Z 9ff218827ef0: Verifying Checksum
2023-01-11T20:43:44.1875287Z 9ff218827ef0: Download complete
2023-01-11T20:43:44.2714116Z bd453ee44627: Verifying Checksum
2023-01-11T20:43:44.2714565Z bd453ee44627: Download complete
2023-01-11T20:43:44.3652534Z e999c29eb8b9: Download complete
2023-01-11T20:43:44.4394093Z 566e975723c3: Download complete
2023-01-11T20:43:44.5920924Z 1072a14c5615: Download complete
2023-01-11T20:43:44.6767966Z 9ae70c90b543: Verifying Checksum
2023-01-11T20:43:44.6768385Z 9ae70c90b543: Download complete
2023-01-11T20:43:44.7476227Z e6f8bf185266: Verifying Checksum
2023-01-11T20:43:44.7503146Z e6f8bf185266: Download complete
2023-01-11T20:43:44.8399203Z 01ff2898de6c: Verifying Checksum
2023-01-11T20:43:44.8399649Z 01ff2898de6c: Download complete
2023-01-11T20:43:44.9372060Z 98f07e351fe7: Verifying Checksum
2023-01-11T20:43:44.9372742Z 98f07e351fe7: Download complete
2023-01-11T20:43:45.0244025Z e426b18803bb: Verifying Checksum
2023-01-11T20:43:45.0244474Z e426b18803bb: Download complete
2023-01-11T20:43:45.1030951Z da3dd65b2531: Verifying Checksum
2023-01-11T20:43:45.1031311Z da3dd65b2531: Download complete
2023-01-11T20:43:45.1693174Z 5c3ec8735bee: Download complete
2023-01-11T20:43:45.2456799Z b616c89406f0: Download complete
2023-01-11T20:43:45.7344361Z f48f88ff2185: Verifying Checksum
2023-01-11T20:43:45.7344794Z f48f88ff2185: Download complete
2023-01-11T20:43:45.8078724Z 46e0cb4c14f2: Verifying Checksum
2023-01-11T20:43:45.8079331Z 46e0cb4c14f2: Download complete
2023-01-11T20:43:45.8922930Z 300515e4431f: Verifying Checksum
2023-01-11T20:43:45.8923391Z 300515e4431f: Download complete
2023-01-11T20:43:45.9697934Z 9df79d1a6e27: Verifying Checksum
2023-01-11T20:43:45.9698424Z 9df79d1a6e27: Download complete
2023-01-11T20:43:46.0474343Z 3cda6364e15e: Verifying Checksum
2023-01-11T20:43:46.0475057Z 3cda6364e15e: Download complete
2023-01-11T20:43:46.2910660Z 7fc4121e1952: Verifying Checksum
2023-01-11T20:43:46.2911093Z 7fc4121e1952: Download complete
2023-01-11T20:43:46.3742434Z 676dcfb6d5c2: Verifying Checksum
2023-01-11T20:43:46.3742873Z 676dcfb6d5c2: Download complete
2023-01-11T20:43:46.3939335Z 0effc56e2dd9: Verifying Checksum
2023-01-11T20:43:46.3939809Z 0effc56e2dd9: Download complete
2023-01-11T20:43:46.4439924Z 130a71e99e46: Verifying Checksum
2023-01-11T20:43:46.4440377Z 130a71e99e46: Download complete
2023-01-11T20:43:46.5426197Z 6cce994e24e8: Verifying Checksum
2023-01-11T20:43:46.5426909Z 6cce994e24e8: Download complete
2023-01-11T20:43:46.6246296Z b37b5d679247: Verifying Checksum
2023-01-11T20:43:46.6246659Z b37b5d679247: Download complete
2023-01-11T20:43:46.7095577Z 86ad3a500f17: Verifying Checksum
2023-01-11T20:43:46.7096100Z 86ad3a500f17: Download complete
2023-01-11T20:43:46.7828787Z 93e8483d7d76: Verifying Checksum
2023-01-11T20:43:46.7829186Z 93e8483d7d76: Download complete
2023-01-11T20:43:46.8723057Z ad1ecb94a573: Verifying Checksum
2023-01-11T20:43:46.8723516Z ad1ecb94a573: Download complete
2023-01-11T20:43:47.4739765Z a7f9c3a7c21d: Verifying Checksum
2023-01-11T20:43:47.4740203Z a7f9c3a7c21d: Download complete
2023-01-11T20:43:49.1565742Z 932461d264d4: Verifying Checksum
2023-01-11T20:43:49.1566053Z 932461d264d4: Download complete
2023-01-11T20:43:50.3874269Z 77ce1d4f36bf: Pull complete
2023-01-11T20:43:50.5608431Z d39938d48fa2: Pull complete
2023-01-11T20:43:50.7075716Z ae4dac5a8919: Pull complete
2023-01-11T20:43:50.9457656Z 4ff68f5f773b: Pull complete
2023-01-11T20:43:51.1843461Z 47642f12aa0f: Pull complete
2023-01-11T20:43:51.4231312Z 3c02291730dc: Pull complete
2023-01-11T20:43:51.6563712Z c7165c4f5e23: Pull complete
2023-01-11T20:43:51.8893433Z 8357faec0a0e: Pull complete
2023-01-11T20:43:53.2846121Z 21ac276b84eb: Verifying Checksum
2023-01-11T20:43:53.2846399Z 21ac276b84eb: Download complete
2023-01-11T20:43:53.7098145Z 42fee15d0ad3: Pull complete
2023-01-11T20:43:53.8526521Z 246016b9e8c8: Pull complete
2023-01-11T20:43:54.0090464Z 25be21861eed: Pull complete
2023-01-11T20:44:14.1278036Z 21ac276b84eb: Pull complete
2023-01-11T20:44:14.4244912Z bd695a285c41: Pull complete
2023-01-11T20:44:14.6755498Z 9af030ba5896: Pull complete
2023-01-11T20:44:15.4907649Z 0ab7fbe28a03: Pull complete
2023-01-11T20:44:15.7345871Z f5dba516d75a: Pull complete
2023-01-11T20:44:16.0034867Z 944388bd6de3: Pull complete
2023-01-11T20:44:16.1656779Z 32233288f470: Pull complete
2023-01-11T20:44:16.3196014Z 4716e681e05e: Pull complete
2023-01-11T20:44:16.4529748Z 07008cd698a0: Pull complete
2023-01-11T20:44:16.5408822Z 27f046e694b1: Pull complete
2023-01-11T20:44:16.6289848Z 95f903a2e692: Pull complete
2023-01-11T20:44:18.4714134Z 1d465abcef1e: Pull complete
2023-01-11T20:44:18.6138790Z 32fdce95f2af: Pull complete
2023-01-11T20:44:18.7798193Z 38f17f1788cc: Pull complete
2023-01-11T20:44:18.9711463Z fa2049e24ad0: Pull complete
2023-01-11T20:44:19.1071870Z 070d94af58e3: Pull complete
2023-01-11T20:44:19.3081864Z 9ff218827ef0: Pull complete
2023-01-11T20:44:23.8724690Z 0effc56e2dd9: Pull complete
2023-01-11T20:44:24.1304623Z bd453ee44627: Pull complete
2023-01-11T20:44:24.3544452Z e999c29eb8b9: Pull complete
2023-01-11T20:44:24.5821201Z 566e975723c3: Pull complete
2023-01-11T20:44:24.7399994Z 1072a14c5615: Pull complete
2023-01-11T20:44:24.8445696Z 9ae70c90b543: Pull complete
2023-01-11T20:44:24.9279816Z e6f8bf185266: Pull complete
2023-01-11T20:44:25.0249315Z 01ff2898de6c: Pull complete
2023-01-11T20:44:25.1223208Z 98f07e351fe7: Pull complete
2023-01-11T20:44:25.2034889Z e426b18803bb: Pull complete
2023-01-11T20:44:25.2902660Z da3dd65b2531: Pull complete
2023-01-11T20:44:25.3796454Z 5c3ec8735bee: Pull complete
2023-01-11T20:44:25.4804418Z b616c89406f0: Pull complete
2023-01-11T20:44:26.7670658Z f48f88ff2185: Pull complete
2023-01-11T20:44:26.9541530Z 46e0cb4c14f2: Pull complete
2023-01-11T20:44:27.1518446Z 300515e4431f: Pull complete
2023-01-11T20:44:27.3474063Z 9df79d1a6e27: Pull complete
2023-01-11T20:44:27.4658023Z 3cda6364e15e: Pull complete
2023-01-11T20:44:27.8465715Z 7fc4121e1952: Pull complete
2023-01-11T20:44:28.0618890Z 676dcfb6d5c2: Pull complete
2023-01-11T20:44:28.2949070Z 130a71e99e46: Pull complete
2023-01-11T20:44:32.1467369Z 932461d264d4: Pull complete
2023-01-11T20:44:32.3713247Z 6cce994e24e8: Pull complete
2023-01-11T20:44:32.5891160Z b37b5d679247: Pull complete
2023-01-11T20:44:32.7916499Z 86ad3a500f17: Pull complete
2023-01-11T20:44:33.0040835Z 93e8483d7d76: Pull complete
2023-01-11T20:44:33.2080893Z ad1ecb94a573: Pull complete
2023-01-11T20:44:34.8146695Z a7f9c3a7c21d: Pull complete
2023-01-11T20:44:34.9243410Z Digest: sha256:d201d5d219c1681863b90b8d50e8b2382dd0a533dc69c772527c8255937144a4
2023-01-11T20:44:34.9752182Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T20:44:34.9994535Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T20:44:35.0046923Z ##[group]Run python3 -m pip install psutil==5.9.1
2023-01-11T20:44:35.0047204Z python3 -m pip install psutil==5.9.1
2023-01-11T20:44:35.0047430Z python3 -m pip install pynvml==11.4.1
2023-01-11T20:44:35.0047688Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2023-01-11T20:44:35.0047972Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2023-01-11T20:44:35.0059028Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T20:44:35.0059237Z env:
2023-01-11T20:44:35.0059416Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:44:35.0059608Z ##[endgroup]
2023-01-11T20:44:35.2186607Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T20:44:35.2373659Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1)
2023-01-11T20:44:35.7498130Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T20:44:35.7685226Z Requirement already satisfied: pynvml==11.4.1 in /home/ec2-user/.local/lib/python3.7/site-packages (11.4.1)
2023-01-11T20:44:36.0775096Z Prepare all required actions
2023-01-11T20:44:36.0775462Z Getting action download info
2023-01-11T20:44:36.2663456Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea)
2023-01-11T20:44:36.5189247Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a)
2023-01-11T20:44:36.6474601Z ##[group]Run ./.github/actions/download-build-artifacts
2023-01-11T20:44:36.6474831Z with:
2023-01-11T20:44:36.6475018Z   name: linux-focal-py3.7-gcc7
2023-01-11T20:44:36.6475190Z env:
2023-01-11T20:44:36.6475361Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:44:36.6475544Z ##[endgroup]
2023-01-11T20:44:36.6500698Z ##[group]Run seemethere/download-artifact-s3@v4
2023-01-11T20:44:36.6500936Z with:
2023-01-11T20:44:36.6501109Z   name: linux-focal-py3.7-gcc7
2023-01-11T20:44:36.6501433Z   s3-bucket: gha-artifacts
2023-01-11T20:44:36.6501625Z   region: us-east-1
2023-01-11T20:44:36.6501776Z env:
2023-01-11T20:44:36.6501983Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:44:36.6502155Z ##[endgroup]
2023-01-11T20:44:37.0492766Z Found 1 objects with prefix pytorch/pytorch/3896099317/linux-focal-py3.7-gcc7/
2023-01-11T20:44:37.0493233Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2023-01-11T20:44:40.1755900Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2023-01-11T20:44:40.1756151Z 
2023-01-11T20:44:40.1768742Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2023-01-11T20:44:40.1776874Z Artifact download has finished successfully
2023-01-11T20:44:40.1871799Z ##[group]Run unzip -o artifacts.zip
2023-01-11T20:44:40.1872040Z unzip -o artifacts.zip
2023-01-11T20:44:40.1883223Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T20:44:40.1883430Z env:
2023-01-11T20:44:40.1883610Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:44:40.1883802Z ##[endgroup]
2023-01-11T20:44:40.1984848Z Archive:  artifacts.zip
2023-01-11T20:44:40.1985650Z    creating: dist/
2023-01-11T20:44:40.9612043Z   inflating: dist/torch-2.0.0a0+git8419ddd-cp37-cp37m-linux_x86_64.whl  
2023-01-11T20:44:40.9612416Z    creating: build/custom_test_artifacts/
2023-01-11T20:44:40.9612738Z    creating: build/custom_test_artifacts/custom-op-build/
2023-01-11T20:44:40.9613104Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2023-01-11T20:44:40.9614634Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2023-01-11T20:44:40.9615436Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2023-01-11T20:44:40.9616160Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2023-01-11T20:44:40.9616608Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2023-01-11T20:44:40.9617016Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2023-01-11T20:44:40.9617912Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2023-01-11T20:44:40.9619168Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2023-01-11T20:44:40.9619967Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2023-01-11T20:44:40.9620410Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2023-01-11T20:44:40.9621734Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-01-11T20:44:40.9622990Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2023-01-11T20:44:40.9624695Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2023-01-11T20:44:40.9625458Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2023-01-11T20:44:40.9626388Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2023-01-11T20:44:40.9627253Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2023-01-11T20:44:40.9627983Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2023-01-11T20:44:40.9628701Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2023-01-11T20:44:40.9629451Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2023-01-11T20:44:40.9630191Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2023-01-11T20:44:40.9631144Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2023-01-11T20:44:40.9631965Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2023-01-11T20:44:40.9632700Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2023-01-11T20:44:40.9633130Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2023-01-11T20:44:40.9633552Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2023-01-11T20:44:40.9633991Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2023-01-11T20:44:40.9634435Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2023-01-11T20:44:40.9634869Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2023-01-11T20:44:40.9635299Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2023-01-11T20:44:40.9648077Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2023-01-11T20:44:40.9737959Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2023-01-11T20:44:40.9738748Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2023-01-11T20:44:40.9739571Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2023-01-11T20:44:40.9740473Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2023-01-11T20:44:40.9741300Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2023-01-11T20:44:40.9742112Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2023-01-11T20:44:40.9742738Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2023-01-11T20:44:40.9743193Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2023-01-11T20:44:40.9743648Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2023-01-11T20:44:40.9744101Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2023-01-11T20:44:40.9744540Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2023-01-11T20:44:40.9757923Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2023-01-11T20:44:40.9824406Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2023-01-11T20:44:40.9825289Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-01-11T20:44:40.9826272Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2023-01-11T20:44:40.9827088Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2023-01-11T20:44:40.9827642Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2023-01-11T20:44:40.9828168Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2023-01-11T20:44:40.9828873Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2023-01-11T20:44:40.9829558Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2023-01-11T20:44:40.9830117Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2023-01-11T20:44:40.9902273Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2023-01-11T20:44:40.9952026Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2023-01-11T20:44:40.9952820Z    creating: build/custom_test_artifacts/jit-hook-build/
2023-01-11T20:44:40.9953170Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2023-01-11T20:44:40.9955035Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2023-01-11T20:44:40.9955749Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2023-01-11T20:44:40.9956477Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2023-01-11T20:44:40.9956893Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2023-01-11T20:44:40.9957294Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2023-01-11T20:44:40.9958068Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2023-01-11T20:44:40.9959307Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2023-01-11T20:44:40.9960089Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2023-01-11T20:44:40.9960515Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2023-01-11T20:44:40.9961824Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-01-11T20:44:40.9963060Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2023-01-11T20:44:40.9964437Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2023-01-11T20:44:40.9965266Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2023-01-11T20:44:40.9966292Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2023-01-11T20:44:40.9967158Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2023-01-11T20:44:40.9967744Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2023-01-11T20:44:40.9968365Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2023-01-11T20:44:40.9969113Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2023-01-11T20:44:40.9969848Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2023-01-11T20:44:40.9970651Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2023-01-11T20:44:40.9971649Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2023-01-11T20:44:40.9972475Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2023-01-11T20:44:40.9973273Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2023-01-11T20:44:40.9973739Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2023-01-11T20:44:40.9974261Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2023-01-11T20:44:40.9974711Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2023-01-11T20:44:40.9975156Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2023-01-11T20:44:40.9975665Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2023-01-11T20:44:40.9988788Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2023-01-11T20:44:41.0039908Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2023-01-11T20:44:41.0040786Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-01-11T20:44:41.0041785Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2023-01-11T20:44:41.0042534Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2023-01-11T20:44:41.0043069Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2023-01-11T20:44:41.0043654Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2023-01-11T20:44:41.0044360Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2023-01-11T20:44:41.0044998Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2023-01-11T20:44:41.0045462Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2023-01-11T20:44:41.0084408Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2023-01-11T20:44:41.0085078Z    creating: build/custom_test_artifacts/custom-backend-build/
2023-01-11T20:44:41.0085443Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2023-01-11T20:44:41.0087436Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2023-01-11T20:44:41.0088204Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2023-01-11T20:44:41.0088854Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2023-01-11T20:44:41.0089293Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2023-01-11T20:44:41.0089726Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2023-01-11T20:44:41.0090718Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2023-01-11T20:44:41.0091738Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2023-01-11T20:44:41.0092550Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2023-01-11T20:44:41.0092987Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2023-01-11T20:44:41.0094269Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-01-11T20:44:41.0095576Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2023-01-11T20:44:41.0097012Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2023-01-11T20:44:41.0097884Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2023-01-11T20:44:41.0098939Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2023-01-11T20:44:41.0099837Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2023-01-11T20:44:41.0100601Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2023-01-11T20:44:41.0101571Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2023-01-11T20:44:41.0102335Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2023-01-11T20:44:41.0103133Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2023-01-11T20:44:41.0103671Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2023-01-11T20:44:41.0104549Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2023-01-11T20:44:41.0105425Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2023-01-11T20:44:41.0106061Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2023-01-11T20:44:41.0106604Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2023-01-11T20:44:41.0107065Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2023-01-11T20:44:41.0107547Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2023-01-11T20:44:41.0108022Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2023-01-11T20:44:41.0108681Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2023-01-11T20:44:41.0109402Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2023-01-11T20:44:41.0226610Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2023-01-11T20:44:41.0227484Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2023-01-11T20:44:41.0228390Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2023-01-11T20:44:41.0229315Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2023-01-11T20:44:41.0230226Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2023-01-11T20:44:41.0231075Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2023-01-11T20:44:41.0231562Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2023-01-11T20:44:41.0232027Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2023-01-11T20:44:41.0232510Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2023-01-11T20:44:41.0232999Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2023-01-11T20:44:41.0233477Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2023-01-11T20:44:41.0246531Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2023-01-11T20:44:41.0293474Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2023-01-11T20:44:41.0294400Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-01-11T20:44:41.0295336Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2023-01-11T20:44:41.0296137Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2023-01-11T20:44:41.0296762Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2023-01-11T20:44:41.0297535Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2023-01-11T20:44:41.0298282Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2023-01-11T20:44:41.0298987Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2023-01-11T20:44:41.0299388Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2023-01-11T20:44:41.0393117Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2023-01-11T20:44:41.0429515Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2023-01-11T20:44:41.0429994Z    creating: build/lib/
2023-01-11T20:44:41.0431275Z   inflating: build/lib/libclog.a     
2023-01-11T20:44:41.0485153Z   inflating: build/lib/libgtest.a    
2023-01-11T20:44:41.0493171Z   inflating: build/lib/libpthreadpool.a  
2023-01-11T20:44:41.0500230Z   inflating: build/lib/libittnotify.a  
2023-01-11T20:44:41.0575330Z   inflating: build/lib/libbenchmark.a  
2023-01-11T20:44:41.0657577Z   inflating: build/lib/libprotobuf-lite.a  
2023-01-11T20:44:41.0682284Z   inflating: build/lib/libtensorpipe_uv.a  
2023-01-11T20:44:41.0778480Z   inflating: build/lib/libgloo.a     
2023-01-11T20:44:41.0838220Z   inflating: build/lib/libasmjit.a   
2023-01-11T20:44:41.1251997Z   inflating: build/lib/libprotobuf.a  
2023-01-11T20:44:41.1277027Z   inflating: build/lib/libfmt.a      
2023-01-11T20:44:41.1277627Z   inflating: build/lib/libfoxi_loader.a  
2023-01-11T20:44:41.1341834Z   inflating: build/lib/libc10.so     
2023-01-11T20:44:41.1342976Z   inflating: build/lib/libtorch_global_deps.so  
2023-01-11T20:44:41.1350327Z   inflating: build/lib/libcpuinfo.a  
2023-01-11T20:44:41.1357354Z   inflating: build/lib/libcpuinfo_internals.a  
2023-01-11T20:44:41.1369484Z   inflating: build/lib/libqnnpack.a  
2023-01-11T20:44:41.1387441Z   inflating: build/lib/libpytorch_qnnpack.a  
2023-01-11T20:44:41.1831230Z   inflating: build/lib/libprotoc.a   
2023-01-11T20:44:41.1832507Z   inflating: build/lib/libnnpack_reference_layers.a  
2023-01-11T20:44:41.1832959Z   inflating: build/lib/libgtest_main.a  
2023-01-11T20:44:41.1846731Z   inflating: build/lib/libgmock.a    
2023-01-11T20:44:41.1847144Z   inflating: build/lib/libbenchmark_main.a  
2023-01-11T20:44:41.9410758Z   inflating: build/lib/libdnnl.a     
2023-01-11T20:44:41.9427759Z   inflating: build/lib/libnnpack.a   
2023-01-11T20:44:41.9933442Z   inflating: build/lib/libtensorpipe.a  
2023-01-11T20:44:41.9933784Z   inflating: build/lib/libgmock_main.a  
2023-01-11T20:44:42.1118942Z   inflating: build/lib/libfbgemm.a   
2023-01-11T20:44:42.2010181Z   inflating: build/lib/libdnnl_graph.a  
2023-01-11T20:44:42.2239224Z   inflating: build/lib/libkineto.a   
2023-01-11T20:44:42.2274989Z   inflating: build/lib/libcaffe2_protos.a  
2023-01-11T20:44:42.2313862Z   inflating: build/lib/libonnx_proto.a  
2023-01-11T20:44:42.2424732Z   inflating: build/lib/libXNNPACK.a  
2023-01-11T20:44:42.2952111Z   inflating: build/lib/libonnx.a     
2023-01-11T20:44:44.1386904Z   inflating: build/lib/libtorch_cpu.so  
2023-01-11T20:44:44.1387341Z   inflating: build/lib/libtorch.so   
2023-01-11T20:44:44.1395703Z   inflating: build/lib/libunbox_lib.a  
2023-01-11T20:44:44.1399817Z   inflating: build/lib/libshm.so     
2023-01-11T20:44:44.1419057Z   inflating: build/lib/libjitbackend_test.so  
2023-01-11T20:44:44.1466931Z   inflating: build/lib/libtorchbind_test.so  
2023-01-11T20:44:44.1491355Z   inflating: build/lib/libbackend_with_compiler.so  
2023-01-11T20:44:44.2841798Z   inflating: build/lib/libtorch_python.so  
2023-01-11T20:44:44.2873285Z   inflating: build/lib/libnnapi_backend.so  
2023-01-11T20:44:44.2873581Z    creating: build/bin/
2023-01-11T20:44:44.2873791Z    creating: build/bin/CMakeFiles/
2023-01-11T20:44:44.2874157Z   inflating: build/bin/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-01-11T20:44:44.2874529Z  extracting: build/bin/CMakeFiles/progress.marks  
2023-01-11T20:44:44.2877178Z   inflating: build/bin/thread_init_test  
2023-01-11T20:44:44.2877926Z   inflating: build/bin/Makefile      
2023-01-11T20:44:44.2945087Z   inflating: build/bin/tensor_iterator_test  
2023-01-11T20:44:44.2945409Z   inflating: build/bin/cmake_install.cmake  
2023-01-11T20:44:44.2993039Z   inflating: build/bin/type_ptr_test  
2023-01-11T20:44:44.2993569Z   inflating: build/bin/CTestTestfile.cmake  
2023-01-11T20:44:44.3038404Z   inflating: build/bin/c10_DeviceGuard_test  
2023-01-11T20:44:44.3081050Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2023-01-11T20:44:44.3124926Z   inflating: build/bin/c10_Device_test  
2023-01-11T20:44:44.3166705Z   inflating: build/bin/c10_StreamGuard_test  
2023-01-11T20:44:44.3217412Z   inflating: build/bin/c10_DispatchKeySet_test  
2023-01-11T20:44:44.3260849Z   inflating: build/bin/c10_SymInt_test  
2023-01-11T20:44:44.3308916Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2023-01-11T20:44:44.3357177Z   inflating: build/bin/c10_InlineStreamGuard_test  
2023-01-11T20:44:44.3406456Z   inflating: build/bin/c10_SizesAndStrides_test  
2023-01-11T20:44:44.3448066Z   inflating: build/bin/c10_Array_test  
2023-01-11T20:44:44.3494379Z   inflating: build/bin/c10_Bitset_test  
2023-01-11T20:44:44.3538793Z   inflating: build/bin/c10_C++17_test  
2023-01-11T20:44:44.3580493Z   inflating: build/bin/c10_ConstexprCrc_test  
2023-01-11T20:44:44.3622866Z   inflating: build/bin/c10_DeadlockDetection_test  
2023-01-11T20:44:44.3665965Z   inflating: build/bin/c10_Half_test  
2023-01-11T20:44:44.3715166Z   inflating: build/bin/c10_LeftRight_test  
2023-01-11T20:44:44.3769168Z   inflating: build/bin/c10_Metaprogramming_test  
2023-01-11T20:44:44.3893176Z   inflating: build/bin/c10_SmallVectorTest  
2023-01-11T20:44:44.3936883Z   inflating: build/bin/c10_Synchronized_test  
2023-01-11T20:44:44.3986001Z   inflating: build/bin/c10_ThreadLocal_test  
2023-01-11T20:44:44.4031855Z   inflating: build/bin/c10_TypeIndex_test  
2023-01-11T20:44:44.4075697Z   inflating: build/bin/c10_TypeList_test  
2023-01-11T20:44:44.4117404Z   inflating: build/bin/c10_TypeTraits_test  
2023-01-11T20:44:44.4162865Z   inflating: build/bin/c10_accumulate_test  
2023-01-11T20:44:44.4211013Z   inflating: build/bin/c10_complex_math_test  
2023-01-11T20:44:44.4258491Z   inflating: build/bin/c10_complex_test  
2023-01-11T20:44:44.4307204Z   inflating: build/bin/c10_bfloat16_test  
2023-01-11T20:44:44.4400619Z   inflating: build/bin/c10_either_test  
2023-01-11T20:44:44.4444196Z   inflating: build/bin/c10_flags_test  
2023-01-11T20:44:44.4490252Z   inflating: build/bin/c10_exception_test  
2023-01-11T20:44:44.4633329Z   inflating: build/bin/c10_intrusive_ptr_test  
2023-01-11T20:44:44.4677166Z   inflating: build/bin/c10_irange_test  
2023-01-11T20:44:44.4726568Z   inflating: build/bin/c10_logging_test  
2023-01-11T20:44:44.4790319Z   inflating: build/bin/c10_optional_test  
2023-01-11T20:44:44.4843802Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2023-01-11T20:44:44.4892154Z   inflating: build/bin/c10_registry_test  
2023-01-11T20:44:44.4942234Z   inflating: build/bin/c10_string_view_test  
2023-01-11T20:44:44.4987212Z   inflating: build/bin/c10_tempfile_test  
2023-01-11T20:44:44.5036192Z   inflating: build/bin/c10_typeid_test  
2023-01-11T20:44:44.5083951Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2023-01-11T20:44:44.5494449Z   inflating: build/bin/protoc-3.13.0.0  
2023-01-11T20:44:44.5904443Z   inflating: build/bin/protoc        
2023-01-11T20:44:44.6155580Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2023-01-11T20:44:44.6435663Z   inflating: build/bin/vec_test_all_types_AVX2  
2023-01-11T20:44:44.6487380Z   inflating: build/bin/TCPStoreTest  
2023-01-11T20:44:44.6534800Z   inflating: build/bin/FileStoreTest  
2023-01-11T20:44:44.6581779Z   inflating: build/bin/HashStoreTest  
2023-01-11T20:44:44.6638330Z   inflating: build/bin/ProcessGroupGlooTest  
2023-01-11T20:44:44.6640952Z   inflating: build/bin/example_allreduce  
2023-01-11T20:44:44.6687943Z   inflating: build/bin/test_dist_autograd  
2023-01-11T20:44:44.6690078Z   inflating: build/bin/parallel_benchmark  
2023-01-11T20:44:44.6749480Z   inflating: build/bin/test_mobile_nnc  
2023-01-11T20:44:44.6809830Z   inflating: build/bin/test_cpp_rpc  
2023-01-11T20:44:44.6818939Z   inflating: build/bin/aot_model_compiler_test  
2023-01-11T20:44:44.6876539Z   inflating: build/bin/static_runtime_bench  
2023-01-11T20:44:44.6939781Z   inflating: build/bin/Dict_test     
2023-01-11T20:44:44.6985740Z   inflating: build/bin/Dimname_test  
2023-01-11T20:44:44.7041104Z   inflating: build/bin/MaybeOwned_test  
2023-01-11T20:44:44.7090585Z   inflating: build/bin/NamedTensor_test  
2023-01-11T20:44:44.7391678Z   inflating: build/bin/test_lazy     
2023-01-11T20:44:44.7442998Z   inflating: build/bin/apply_utils_test  
2023-01-11T20:44:44.7657726Z   inflating: build/bin/static_runtime_test  
2023-01-11T20:44:44.7708558Z   inflating: build/bin/atest         
2023-01-11T20:44:44.7761268Z   inflating: build/bin/basic         
2023-01-11T20:44:44.7808380Z   inflating: build/bin/broadcast_test  
2023-01-11T20:44:44.7858701Z   inflating: build/bin/cpu_generator_test  
2023-01-11T20:44:44.7904425Z   inflating: build/bin/cpu_profiling_allocator_test  
2023-01-11T20:44:44.7947740Z   inflating: build/bin/dispatch_key_set_test  
2023-01-11T20:44:44.8023794Z   inflating: build/bin/cpu_rng_test  
2023-01-11T20:44:44.8066762Z   inflating: build/bin/dlconvertor_test  
2023-01-11T20:44:44.8117379Z   inflating: build/bin/extension_backend_test  
2023-01-11T20:44:44.8165593Z   inflating: build/bin/half_test     
2023-01-11T20:44:44.8247224Z   inflating: build/bin/ivalue_test   
2023-01-11T20:44:44.8289802Z   inflating: build/bin/lazy_tensor_test  
2023-01-11T20:44:44.8336974Z   inflating: build/bin/math_kernel_test  
2023-01-11T20:44:44.8383629Z   inflating: build/bin/memory_format_test  
2023-01-11T20:44:44.8430360Z   inflating: build/bin/memory_overlapping_test  
2023-01-11T20:44:44.8478810Z   inflating: build/bin/native_test   
2023-01-11T20:44:44.8524676Z   inflating: build/bin/mobile_memory_cleanup  
2023-01-11T20:44:44.8568775Z   inflating: build/bin/operator_name_test  
2023-01-11T20:44:44.8612495Z   inflating: build/bin/operators_test  
2023-01-11T20:44:44.8658415Z   inflating: build/bin/packedtensoraccessor_test  
2023-01-11T20:44:44.8708500Z   inflating: build/bin/quantized_test  
2023-01-11T20:44:44.9721927Z   inflating: build/bin/test_api      
2023-01-11T20:44:44.9778483Z   inflating: build/bin/pow_test      
2023-01-11T20:44:44.9821285Z   inflating: build/bin/reduce_ops_test  
2023-01-11T20:44:44.9865624Z   inflating: build/bin/reportMemoryUsage_test  
2023-01-11T20:44:44.9914715Z   inflating: build/bin/scalar_tensor_test  
2023-01-11T20:44:44.9964386Z   inflating: build/bin/scalar_test   
2023-01-11T20:44:45.0010960Z   inflating: build/bin/stride_properties_test  
2023-01-11T20:44:45.0059715Z   inflating: build/bin/test_parallel  
2023-01-11T20:44:45.0102609Z   inflating: build/bin/variant_test  
2023-01-11T20:44:45.0155392Z   inflating: build/bin/type_test     
2023-01-11T20:44:45.0201263Z   inflating: build/bin/undefined_tensor_test  
2023-01-11T20:44:45.0202563Z   inflating: build/bin/verify_api_visibility  
2023-01-11T20:44:45.0262287Z   inflating: build/bin/legacy_vmap_test  
2023-01-11T20:44:45.0306635Z   inflating: build/bin/weakref_test  
2023-01-11T20:44:45.0350829Z   inflating: build/bin/wrapdim_test  
2023-01-11T20:44:45.0402636Z   inflating: build/bin/IListRef_test  
2023-01-11T20:44:45.0444965Z   inflating: build/bin/xla_tensor_test  
2023-01-11T20:44:45.0538330Z   inflating: build/bin/List_test     
2023-01-11T20:44:45.0642598Z   inflating: build/bin/kernel_function_legacy_test  
2023-01-11T20:44:45.0725616Z   inflating: build/bin/kernel_function_test  
2023-01-11T20:44:45.0782372Z   inflating: build/bin/KernelFunction_test  
2023-01-11T20:44:45.0892134Z   inflating: build/bin/kernel_lambda_legacy_test  
2023-01-11T20:44:45.0981171Z   inflating: build/bin/kernel_lambda_test  
2023-01-11T20:44:45.1033140Z   inflating: build/bin/kernel_stackbased_test  
2023-01-11T20:44:45.1077165Z   inflating: build/bin/CppSignature_test  
2023-01-11T20:44:45.1159202Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2023-01-11T20:44:45.1201286Z   inflating: build/bin/op_allowlist_test  
2023-01-11T20:44:45.1248168Z   inflating: build/bin/inline_container_test  
2023-01-11T20:44:45.1492452Z   inflating: build/bin/op_registration_test  
2023-01-11T20:44:45.1541104Z   inflating: build/bin/backend_fallback_test  
2023-01-11T20:44:45.1590576Z   inflating: build/bin/test_edge_op_registration  
2023-01-11T20:44:45.1595347Z   inflating: build/bin/torch_shm_manager  
2023-01-11T20:44:45.1609347Z   inflating: build/bin/tutorial_tensorexpr  
2023-01-11T20:44:45.2288877Z   inflating: build/bin/test_tensorexpr  
2023-01-11T20:44:45.2781718Z   inflating: build/bin/test_jit      
2023-01-11T20:44:45.2783168Z   inflating: .pytorch-test-times.json  
2023-01-11T20:44:45.2810769Z ##[group]Run df -H
2023-01-11T20:44:45.2811231Z df -H
2023-01-11T20:44:45.2822213Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T20:44:45.2822427Z env:
2023-01-11T20:44:45.2822603Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:44:45.2822788Z ##[endgroup]
2023-01-11T20:44:45.3070428Z Filesystem      Size  Used Avail Use% Mounted on
2023-01-11T20:44:45.3070845Z devtmpfs        8.2G     0  8.2G   0% /dev
2023-01-11T20:44:45.3071237Z tmpfs           8.2G  9.9M  8.2G   1% /dev/shm
2023-01-11T20:44:45.3071540Z tmpfs           8.2G  431k  8.2G   1% /run
2023-01-11T20:44:45.3071994Z tmpfs           8.2G     0  8.2G   0% /sys/fs/cgroup
2023-01-11T20:44:45.3073141Z /dev/nvme0n1p1  162G   18G  144G  11% /
2023-01-11T20:44:45.3121263Z ##[group]Run .github/scripts/parse_ref.py
2023-01-11T20:44:45.3121510Z .github/scripts/parse_ref.py
2023-01-11T20:44:45.3131964Z shell: /usr/bin/bash -e {0}
2023-01-11T20:44:45.3132135Z env:
2023-01-11T20:44:45.3132306Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:44:45.3132501Z ##[endgroup]
2023-01-11T20:44:45.3361271Z ##[group]Run set -x
2023-01-11T20:44:45.3361532Z set -x
2023-01-11T20:44:45.3361697Z 
2023-01-11T20:44:45.3361893Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2023-01-11T20:44:45.3362137Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2023-01-11T20:44:45.3362394Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2023-01-11T20:44:45.3362626Z   TEST_COMMAND=.jenkins/onnx/test.sh
2023-01-11T20:44:45.3362811Z else
2023-01-11T20:44:45.3363010Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2023-01-11T20:44:45.3363346Z fi
2023-01-11T20:44:45.3363491Z 
2023-01-11T20:44:45.3363723Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2023-01-11T20:44:45.3363957Z 
2023-01-11T20:44:45.3364170Z # sanitize the input commit message and PR body here:
2023-01-11T20:44:45.3364368Z #
2023-01-11T20:44:45.3364644Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment
2023-01-11T20:44:45.3365019Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028
2023-01-11T20:44:45.3365315Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}"
2023-01-11T20:44:45.3365541Z PR_BODY="${PR_BODY//[$'\n\r']}"
2023-01-11T20:44:45.3365724Z 
2023-01-11T20:44:45.3365970Z # then trim all special characters like single and double quotes to avoid unescaped inputs to
2023-01-11T20:44:45.3366244Z # wreak havoc internally
2023-01-11T20:44:45.3366483Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}"
2023-01-11T20:44:45.3366721Z export PR_BODY="${PR_BODY//[\'\"]}"
2023-01-11T20:44:45.3366894Z 
2023-01-11T20:44:45.3367117Z # detached container should get cleaned up by teardown_ec2_linux
2023-01-11T20:44:45.3367408Z # TODO: Stop building test binaries as part of the build phase
2023-01-11T20:44:45.3367667Z # Used for GPU_FLAG since that doesn't play nice
2023-01-11T20:44:45.3367903Z # shellcheck disable=SC2086,SC2090
2023-01-11T20:44:45.3368122Z container_name=$(docker run \
2023-01-11T20:44:45.3368307Z   ${GPU_FLAG:-} \
2023-01-11T20:44:45.3368498Z   -e BUILD_ENVIRONMENT \
2023-01-11T20:44:45.3368692Z   -e PR_NUMBER \
2023-01-11T20:44:45.3368886Z   -e GITHUB_ACTIONS \
2023-01-11T20:44:45.3369062Z   -e BASE_SHA \
2023-01-11T20:44:45.3369244Z   -e BRANCH \
2023-01-11T20:44:45.3369422Z   -e SHA1 \
2023-01-11T20:44:45.3369597Z   -e AWS_DEFAULT_REGION \
2023-01-11T20:44:45.3369799Z   -e IN_WHEEL_TEST \
2023-01-11T20:44:45.3369991Z   -e SHARD_NUMBER \
2023-01-11T20:44:45.3370170Z   -e TEST_CONFIG \
2023-01-11T20:44:45.3370364Z   -e NUM_TEST_SHARDS \
2023-01-11T20:44:45.3370552Z   -e PR_BODY \
2023-01-11T20:44:45.3370989Z   -e COMMIT_MESSAGES \
2023-01-11T20:44:45.3371204Z   -e CONTINUE_THROUGH_ERROR \
2023-01-11T20:44:45.3371425Z   -e PYTORCH_RETRY_TEST_CASES \
2023-01-11T20:44:45.3371643Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2023-01-11T20:44:45.3371852Z   -e PR_LABELS \
2023-01-11T20:44:45.3372063Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2023-01-11T20:44:45.3372275Z   -e SCCACHE_BUCKET \
2023-01-11T20:44:45.3372463Z   -e SCCACHE_S3_KEY_PREFIX \
2023-01-11T20:44:45.3372659Z   -e XLA_CUDA \
2023-01-11T20:44:45.3372869Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2023-01-11T20:44:45.3373091Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2023-01-11T20:44:45.3373332Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2023-01-11T20:44:45.3373587Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2023-01-11T20:44:45.3373809Z   --ulimit stack=10485760:83886080 \
2023-01-11T20:44:45.3374111Z   --security-opt seccomp=unconfined \
2023-01-11T20:44:45.3374336Z   --cap-add=SYS_PTRACE \
2023-01-11T20:44:45.3374519Z   --ipc=host \
2023-01-11T20:44:45.3374713Z   --shm-size="${SHM_SIZE}" \
2023-01-11T20:44:45.3374904Z   --tty \
2023-01-11T20:44:45.3375076Z   --detach \
2023-01-11T20:44:45.3375255Z   --name="${container_name}" \
2023-01-11T20:44:45.3375520Z   --user jenkins \
2023-01-11T20:44:45.3375754Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2023-01-11T20:44:45.3375989Z   -w /var/lib/jenkins/workspace \
2023-01-11T20:44:45.3376190Z   "${DOCKER_IMAGE}"
2023-01-11T20:44:45.3376359Z )
2023-01-11T20:44:45.3376563Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2023-01-11T20:44:45.3376887Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2023-01-11T20:44:45.3386926Z shell: /usr/bin/bash -e {0}
2023-01-11T20:44:45.3387111Z env:
2023-01-11T20:44:45.3387286Z   GIT_DEFAULT_BRANCH: master
2023-01-11T20:44:45.3387505Z   BUILD_ENVIRONMENT: linux-focal-py3.7-gcc7
2023-01-11T20:44:45.3387719Z   PR_NUMBER: 91627
2023-01-11T20:44:45.3387897Z   BRANCH: pull/91627
2023-01-11T20:44:45.3388100Z   SHA1: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T20:44:45.3388346Z   BASE_SHA: db2a237763eb8693a20788be94f8c192e762baa8
2023-01-11T20:44:45.3388570Z   PYTORCH_RETRY_TEST_CASES: 1
2023-01-11T20:44:45.3388764Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2023-01-11T20:44:45.3388965Z   TEST_CONFIG: distributed
2023-01-11T20:44:45.3389146Z   SHARD_NUMBER: 2
2023-01-11T20:44:45.3389315Z   NUM_TEST_SHARDS: 2
2023-01-11T20:44:45.3389504Z   PR_BODY: Fixes #91003 


cc @ezyang @gchanan
2023-01-11T20:44:45.3389716Z   CONTINUE_THROUGH_ERROR: False
2023-01-11T20:44:45.3389958Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2023-01-11T20:44:45.3390184Z   SCCACHE_S3_KEY_PREFIX: pull
2023-01-11T20:44:45.3390370Z   SHM_SIZE: 1g
2023-01-11T20:44:45.3390708Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T20:44:45.3391019Z   XLA_CUDA: 
2023-01-11T20:44:45.3391274Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2023-01-11T20:44:45.3391550Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2023-01-11T20:44:45.3391767Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2023-01-11T20:44:45.3391949Z ##[endgroup]
2023-01-11T20:44:45.3414628Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2023-01-11T20:44:45.3415259Z + [[ linux-focal-py3.7-gcc7 == *onnx* ]]
2023-01-11T20:44:45.3415694Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2023-01-11T20:44:45.3418210Z ++ git cherry -v origin/master
2023-01-11T20:44:45.3852084Z + COMMIT_MESSAGES='+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into '\''input'\''
2023-01-11T20:44:45.3852463Z + 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch
2023-01-11T20:44:45.3853034Z + 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e '\''other'\'' instead of '\''output'\'' in documentation'
2023-01-11T20:44:45.3853792Z + COMMIT_MESSAGES='+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into '\''input'\''+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e '\''other'\'' instead of '\''output'\'' in documentation'
2023-01-11T20:44:45.3854325Z + PR_BODY='Fixes #91003 cc @ezyang @gchanan'
2023-01-11T20:44:45.3855770Z + export 'COMMIT_MESSAGES=+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation'
2023-01-11T20:44:45.3856730Z + COMMIT_MESSAGES='+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation'
2023-01-11T20:44:45.3857229Z + export 'PR_BODY=Fixes #91003 cc @ezyang @gchanan'
2023-01-11T20:44:45.3857509Z + PR_BODY='Fixes #91003 cc @ezyang @gchanan'
2023-01-11T20:44:45.3863473Z +++ nproc --ignore=2
2023-01-11T20:44:45.3873555Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e CONTINUE_THROUGH_ERROR -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=6 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS --env-file=/tmp/github_env_3896099317 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T20:44:56.7176459Z + container_name=16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58
2023-01-11T20:44:56.7176851Z + echo DOCKER_CONTAINER_ID=16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58
2023-01-11T20:44:56.7180911Z ++ echo dist/torch-2.0.0a0+git8419ddd-cp37-cp37m-linux_x86_64.whl
2023-01-11T20:44:56.7182192Z + docker exec -t 16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58 sh -c 'pip install dist/torch-2.0.0a0+git8419ddd-cp37-cp37m-linux_x86_64.whl[opt-einsum] && .jenkins/pytorch/test.sh'
2023-01-11T20:44:57.1081098Z Processing ./dist/torch-2.0.0a0+git8419ddd-cp37-cp37m-linux_x86_64.whl
2023-01-11T20:44:57.4697231Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch==2.0.0a0+git8419ddd) (4.4.0)
2023-01-11T20:44:57.4700542Z Requirement already satisfied: networkx in /opt/conda/lib/python3.7/site-packages (from torch==2.0.0a0+git8419ddd) (2.6.3)
2023-01-11T20:44:57.4704987Z Requirement already satisfied: sympy in /opt/conda/lib/python3.7/site-packages (from torch==2.0.0a0+git8419ddd) (1.10.1)
2023-01-11T20:44:57.4718661Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/lib/python3.7/site-packages (from torch==2.0.0a0+git8419ddd) (3.3.0)
2023-01-11T20:44:57.4779996Z Requirement already satisfied: numpy>=1.7 in /opt/conda/lib/python3.7/site-packages (from opt-einsum>=3.3->torch==2.0.0a0+git8419ddd) (1.18.5)
2023-01-11T20:44:57.4947346Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.7/site-packages (from sympy->torch==2.0.0a0+git8419ddd) (1.2.1)
2023-01-11T20:44:58.2148166Z Installing collected packages: torch
2023-01-11T20:45:03.7790186Z Successfully installed torch-2.0.0a0+git8419ddd
2023-01-11T20:45:03.9077105Z + echo 'Environment variables:'
2023-01-11T20:45:03.9077356Z Environment variables:
2023-01-11T20:45:03.9077520Z + env
2023-01-11T20:45:03.9082800Z INSTALLED_DB=yes
2023-01-11T20:45:03.9084048Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T20:45:03.9084313Z CONTINUE_THROUGH_ERROR=False
2023-01-11T20:45:03.9084574Z BUILD_ENVIRONMENT=linux-focal-py3.7-gcc7
2023-01-11T20:45:03.9084808Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2023-01-11T20:45:03.9085011Z HOSTNAME=16a6623252a2
2023-01-11T20:45:03.9085722Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_0a0d64e7-ea5c-44f9-9f1d-c5fd0b955c07
2023-01-11T20:45:03.9086167Z GITHUB_ACTION=__self
2023-01-11T20:45:03.9086401Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2023-01-11T20:45:03.9086592Z GITHUB_RUN_NUMBER=77928
2023-01-11T20:45:03.9086806Z TEST_CONFIG=distributed
2023-01-11T20:45:03.9087079Z GITHUB_REPOSITORY_OWNER_ID=21003710
2023-01-11T20:45:03.9087403Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2023-01-11T20:45:03.9087896Z GITHUB_TRIGGERING_ACTOR=albanD
2023-01-11T20:45:03.9090130Z GITHUB_REF_TYPE=branch
2023-01-11T20:45:03.9090598Z TORCH_CUDA_ARCH_LIST=Maxwell
2023-01-11T20:45:03.9091212Z BASE_SHA=db2a237763eb8693a20788be94f8c192e762baa8
2023-01-11T20:45:03.9091577Z XLA_CUDA=
2023-01-11T20:45:03.9097520Z ***
2023-01-11T20:45:03.9097866Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2023-01-11T20:45:03.9098237Z GITHUB_REPOSITORY_ID=65600975
2023-01-11T20:45:03.9098560Z GITHUB_ACTIONS=true
2023-01-11T20:45:03.9098931Z SHA1=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T20:45:03.9099357Z GITHUB_SHA=57fc38f02f250896a12b32cfa200a6105a03d09c
2023-01-11T20:45:03.9099853Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/91627/merge
2023-01-11T20:45:03.9100293Z UCC_HOME=/usr
2023-01-11T20:45:03.9100616Z GITHUB_REF=refs/pull/91627/merge
2023-01-11T20:45:03.9100939Z SHARD_NUMBER=2
2023-01-11T20:45:03.9101261Z GITHUB_REF_PROTECTED=false
2023-01-11T20:45:03.9101597Z HOME=/var/lib/jenkins
2023-01-11T20:45:03.9102003Z GITHUB_API_URL=https://api.github.com
2023-01-11T20:45:03.9102407Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2023-01-11T20:45:03.9102850Z INSTALLED_THRIFT=
2023-01-11T20:45:03.9103149Z UCX_COMMIT=
2023-01-11T20:45:03.9103443Z SCCACHE_S3_KEY_PREFIX=pull
2023-01-11T20:45:03.9103778Z NUM_TEST_SHARDS=2
2023-01-11T20:45:03.9104076Z UCX_HOME=/usr
2023-01-11T20:45:03.9104842Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_0a0d64e7-ea5c-44f9-9f1d-c5fd0b955c07
2023-01-11T20:45:03.9105381Z PYTORCH_RETRY_TEST_CASES=1
2023-01-11T20:45:03.9106098Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_0a0d64e7-ea5c-44f9-9f1d-c5fd0b955c07
2023-01-11T20:45:03.9106858Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2023-01-11T20:45:03.9107287Z GITHUB_EVENT_NAME=pull_request
2023-01-11T20:45:03.9107611Z GITHUB_RUN_ID=3896099317
2023-01-11T20:45:03.9108364Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_0a0d64e7-ea5c-44f9-9f1d-c5fd0b955c07
2023-01-11T20:45:03.9108893Z GITHUB_ACTOR=LucaLumetti
2023-01-11T20:45:03.9109220Z PR_NUMBER=91627
2023-01-11T20:45:03.9109525Z DESIRED_CUDA=
2023-01-11T20:45:03.9109814Z GITHUB_RUN_ATTEMPT=2
2023-01-11T20:45:03.9110230Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2023-01-11T20:45:03.9110616Z TERM=xterm
2023-01-11T20:45:03.9110893Z INSTALLED_VISION=yes
2023-01-11T20:45:03.9111210Z BRANCH=pull/91627
2023-01-11T20:45:03.9111536Z OPENSSL_ROOT_DIR=/opt/openssl
2023-01-11T20:45:03.9111867Z CUDA_PATH=/usr/local/cuda
2023-01-11T20:45:03.9112525Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2023-01-11T20:45:03.9113059Z GITHUB_SERVER_URL=https://github.com
2023-01-11T20:45:03.9113401Z UCC_COMMIT=
2023-01-11T20:45:03.9113659Z INSTALLED_ANDROID=
2023-01-11T20:45:03.9113938Z SHLVL=1
2023-01-11T20:45:03.9114192Z MAX_JOBS=6
2023-01-11T20:45:03.9114479Z GITHUB_ACTOR_ID=7543386
2023-01-11T20:45:03.9115482Z COMMIT_MESSAGES=+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation
2023-01-11T20:45:03.9116443Z GITHUB_WORKFLOW_SHA=57fc38f02f250896a12b32cfa200a6105a03d09c
2023-01-11T20:45:03.9116832Z GITHUB_REF_NAME=91627/merge
2023-01-11T20:45:03.9117433Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2023-01-11T20:45:03.9117893Z GITHUB_JOB=test
2023-01-11T20:45:03.9118222Z GITHUB_REPOSITORY=pytorch/pytorch
2023-01-11T20:45:03.9118561Z GITHUB_RETENTION_DAYS=90
2023-01-11T20:45:03.9118885Z OPENSSL_DIR=/opt/openssl
2023-01-11T20:45:03.9119223Z GITHUB_ACTION_REPOSITORY=
2023-01-11T20:45:03.9119761Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T20:45:03.9120428Z GITHUB_BASE_REF=master
2023-01-11T20:45:03.9120732Z CI=true
2023-01-11T20:45:03.9121038Z GITHUB_REPOSITORY_OWNER=pytorch
2023-01-11T20:45:03.9121396Z INSTALLED_PROTOBUF=yes
2023-01-11T20:45:03.9121720Z GITHUB_HEAD_REF=master
2023-01-11T20:45:03.9122010Z GITHUB_ACTION_REF=
2023-01-11T20:45:03.9122529Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2023-01-11T20:45:03.9122940Z GITHUB_WORKFLOW=pull
2023-01-11T20:45:03.9123279Z DEBIAN_FRONTEND=noninteractive
2023-01-11T20:45:03.9124052Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_0a0d64e7-ea5c-44f9-9f1d-c5fd0b955c07
2023-01-11T20:45:03.9124616Z PR_BODY=Fixes #91003 cc @ezyang @gchanan
2023-01-11T20:45:03.9124938Z _=/usr/bin/env
2023-01-11T20:45:03.9125413Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2023-01-11T20:45:03.9251555Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.7/site-packages/torch
2023-01-11T20:45:03.9252048Z + TORCH_BIN_DIR=/opt/conda/lib/python3.7/site-packages/torch/bin
2023-01-11T20:45:03.9252387Z + TORCH_LIB_DIR=/opt/conda/lib/python3.7/site-packages/torch/lib
2023-01-11T20:45:03.9252952Z + TORCH_TEST_DIR=/opt/conda/lib/python3.7/site-packages/torch/test
2023-01-11T20:45:03.9253351Z + BUILD_DIR=build
2023-01-11T20:45:03.9253650Z + BUILD_RENAMED_DIR=build_renamed
2023-01-11T20:45:03.9253991Z + BUILD_BIN_DIR=build/bin
2023-01-11T20:45:03.9254237Z + export VALGRIND=ON
2023-01-11T20:45:03.9254409Z + VALGRIND=ON
2023-01-11T20:45:03.9254590Z + export TORCH_INDUCTOR_INSTALL_GXX=ON
2023-01-11T20:45:03.9254805Z + TORCH_INDUCTOR_INSTALL_GXX=ON
2023-01-11T20:45:03.9255087Z + [[ linux-focal-py3.7-gcc7 == *clang9* ]]
2023-01-11T20:45:03.9255434Z + [[ linux-focal-py3.7-gcc7 != *bazel* ]]
2023-01-11T20:45:03.9257625Z ++ realpath build/custom_test_artifacts
2023-01-11T20:45:03.9264185Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2023-01-11T20:45:03.9271412Z ++ dirname .jenkins/pytorch/test.sh
2023-01-11T20:45:03.9272468Z + source .jenkins/pytorch/common.sh
2023-01-11T20:45:03.9275572Z +++ dirname .jenkins/pytorch/common.sh
2023-01-11T20:45:03.9280758Z ++ source .jenkins/pytorch/common_utils.sh
2023-01-11T20:45:03.9282304Z +++ declare -f -t trap_add
2023-01-11T20:45:03.9288025Z ++ set -ex
2023-01-11T20:45:03.9288471Z ++ [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2023-01-11T20:45:03.9288847Z ++ BUILD_TEST_LIBTORCH=0
2023-01-11T20:45:03.9289281Z + echo 'Environment variables'
2023-01-11T20:45:03.9289513Z Environment variables
2023-01-11T20:45:03.9289674Z + env
2023-01-11T20:45:03.9294629Z INSTALLED_DB=yes
2023-01-11T20:45:03.9295221Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T20:45:03.9295728Z CONTINUE_THROUGH_ERROR=False
2023-01-11T20:45:03.9296187Z BUILD_ENVIRONMENT=linux-focal-py3.7-gcc7
2023-01-11T20:45:03.9296571Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2023-01-11T20:45:03.9296882Z HOSTNAME=16a6623252a2
2023-01-11T20:45:03.9297364Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_0a0d64e7-ea5c-44f9-9f1d-c5fd0b955c07
2023-01-11T20:45:03.9297936Z GITHUB_ACTION=__self
2023-01-11T20:45:03.9298207Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2023-01-11T20:45:03.9298420Z GITHUB_RUN_NUMBER=77928
2023-01-11T20:45:03.9298669Z TEST_CONFIG=distributed
2023-01-11T20:45:03.9298876Z GITHUB_REPOSITORY_OWNER_ID=21003710
2023-01-11T20:45:03.9299179Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2023-01-11T20:45:03.9299401Z GITHUB_TRIGGERING_ACTOR=albanD
2023-01-11T20:45:03.9299596Z GITHUB_REF_TYPE=branch
2023-01-11T20:45:03.9299775Z TORCH_CUDA_ARCH_LIST=Maxwell
2023-01-11T20:45:03.9299998Z BASE_SHA=db2a237763eb8693a20788be94f8c192e762baa8
2023-01-11T20:45:03.9300231Z XLA_CUDA=
2023-01-11T20:45:03.9300516Z ***
2023-01-11T20:45:03.9300767Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2023-01-11T20:45:03.9301074Z GITHUB_REPOSITORY_ID=65600975
2023-01-11T20:45:03.9301314Z GITHUB_ACTIONS=true
2023-01-11T20:45:03.9301583Z SHA1=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T20:45:03.9301910Z GITHUB_SHA=57fc38f02f250896a12b32cfa200a6105a03d09c
2023-01-11T20:45:03.9302421Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/pull.yml@refs/pull/91627/merge
2023-01-11T20:45:03.9302767Z UCC_HOME=/usr
2023-01-11T20:45:03.9303019Z GITHUB_REF=refs/pull/91627/merge
2023-01-11T20:45:03.9303195Z SHARD_NUMBER=2
2023-01-11T20:45:03.9303449Z GITHUB_REF_PROTECTED=false
2023-01-11T20:45:03.9303641Z HOME=/var/lib/jenkins
2023-01-11T20:45:03.9303856Z GITHUB_API_URL=https://api.github.com
2023-01-11T20:45:03.9304078Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2023-01-11T20:45:03.9304277Z INSTALLED_THRIFT=
2023-01-11T20:45:03.9304446Z UCX_COMMIT=
2023-01-11T20:45:03.9304612Z SCCACHE_S3_KEY_PREFIX=pull
2023-01-11T20:45:03.9304797Z NUM_TEST_SHARDS=2
2023-01-11T20:45:03.9304965Z UCX_HOME=/usr
2023-01-11T20:45:03.9305368Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_0a0d64e7-ea5c-44f9-9f1d-c5fd0b955c07
2023-01-11T20:45:03.9305671Z TORCH_INDUCTOR_INSTALL_GXX=ON
2023-01-11T20:45:03.9305870Z PYTORCH_RETRY_TEST_CASES=1
2023-01-11T20:45:03.9306263Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_0a0d64e7-ea5c-44f9-9f1d-c5fd0b955c07
2023-01-11T20:45:03.9309523Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2023-01-11T20:45:03.9309973Z GITHUB_EVENT_NAME=pull_request
2023-01-11T20:45:03.9310314Z GITHUB_RUN_ID=3896099317
2023-01-11T20:45:03.9311009Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_0a0d64e7-ea5c-44f9-9f1d-c5fd0b955c07
2023-01-11T20:45:03.9311307Z GITHUB_ACTOR=LucaLumetti
2023-01-11T20:45:03.9311493Z PR_NUMBER=91627
2023-01-11T20:45:03.9311666Z DESIRED_CUDA=
2023-01-11T20:45:03.9311829Z GITHUB_RUN_ATTEMPT=2
2023-01-11T20:45:03.9312005Z VALGRIND=ON
2023-01-11T20:45:03.9312230Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2023-01-11T20:45:03.9312433Z TERM=xterm
2023-01-11T20:45:03.9312604Z INSTALLED_VISION=yes
2023-01-11T20:45:03.9312781Z BRANCH=pull/91627
2023-01-11T20:45:03.9312955Z OPENSSL_ROOT_DIR=/opt/openssl
2023-01-11T20:45:03.9313158Z CUDA_PATH=/usr/local/cuda
2023-01-11T20:45:03.9313526Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2023-01-11T20:45:03.9313808Z GITHUB_SERVER_URL=https://github.com
2023-01-11T20:45:03.9314006Z UCC_COMMIT=
2023-01-11T20:45:03.9314178Z INSTALLED_ANDROID=
2023-01-11T20:45:03.9314332Z SHLVL=1
2023-01-11T20:45:03.9314486Z MAX_JOBS=6
2023-01-11T20:45:03.9314654Z GITHUB_ACTOR_ID=7543386
2023-01-11T20:45:03.9315102Z COMMIT_MESSAGES=+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation
2023-01-11T20:45:03.9315715Z GITHUB_WORKFLOW_SHA=57fc38f02f250896a12b32cfa200a6105a03d09c
2023-01-11T20:45:03.9316065Z GITHUB_REF_NAME=91627/merge
2023-01-11T20:45:03.9316699Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2023-01-11T20:45:03.9317106Z GITHUB_JOB=test
2023-01-11T20:45:03.9317290Z GITHUB_REPOSITORY=pytorch/pytorch
2023-01-11T20:45:03.9317497Z GITHUB_RETENTION_DAYS=90
2023-01-11T20:45:03.9317705Z OPENSSL_DIR=/opt/openssl
2023-01-11T20:45:03.9317887Z GITHUB_ACTION_REPOSITORY=
2023-01-11T20:45:03.9318202Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T20:45:03.9318488Z GITHUB_BASE_REF=master
2023-01-11T20:45:03.9318649Z CI=true
2023-01-11T20:45:03.9318827Z GITHUB_REPOSITORY_OWNER=pytorch
2023-01-11T20:45:03.9319025Z INSTALLED_PROTOBUF=yes
2023-01-11T20:45:03.9319198Z GITHUB_HEAD_REF=master
2023-01-11T20:45:03.9319379Z GITHUB_ACTION_REF=
2023-01-11T20:45:03.9319656Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2023-01-11T20:45:03.9319887Z GITHUB_WORKFLOW=pull
2023-01-11T20:45:03.9320071Z DEBIAN_FRONTEND=noninteractive
2023-01-11T20:45:03.9320533Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_0a0d64e7-ea5c-44f9-9f1d-c5fd0b955c07
2023-01-11T20:45:03.9320847Z PR_BODY=Fixes #91003 cc @ezyang @gchanan
2023-01-11T20:45:03.9321024Z _=/usr/bin/env
2023-01-11T20:45:03.9321231Z + echo 'Testing pytorch'
2023-01-11T20:45:03.9321411Z Testing pytorch
2023-01-11T20:45:03.9321590Z + export LANG=C.UTF-8
2023-01-11T20:45:03.9321777Z + LANG=C.UTF-8
2023-01-11T20:45:03.9321947Z + PR_NUMBER=91627
2023-01-11T20:45:03.9322126Z + [[ distributed == \d\e\f\a\u\l\t ]]
2023-01-11T20:45:03.9322342Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2023-01-11T20:45:03.9322609Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2023-01-11T20:45:03.9322810Z + [[ distributed == \s\l\o\w ]]
2023-01-11T20:45:03.9323084Z + [[ linux-focal-py3.7-gcc7 == *slow-gradcheck* ]]
2023-01-11T20:45:03.9323366Z + [[ linux-focal-py3.7-gcc7 == *cuda* ]]
2023-01-11T20:45:03.9323617Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2023-01-11T20:45:03.9323831Z + [[ distributed == *crossref* ]]
2023-01-11T20:45:03.9324028Z + [[ distributed == *dynamo* ]]
2023-01-11T20:45:03.9324220Z + [[ distributed == *inductor* ]]
2023-01-11T20:45:03.9324472Z + [[ linux-focal-py3.7-gcc7 == *rocm* ]]
2023-01-11T20:45:03.9324740Z + [[ linux-focal-py3.7-gcc7 != *-bazel-* ]]
2023-01-11T20:45:03.9325004Z + pip_install --user ninja==1.10.2
2023-01-11T20:45:03.9325277Z + pip install --progress-bar off --user ninja==1.10.2
2023-01-11T20:45:04.3210329Z Collecting ninja==1.10.2
2023-01-11T20:45:04.3533566Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2023-01-11T20:45:05.0571027Z Installing collected packages: ninja
2023-01-11T20:45:05.0655410Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2023-01-11T20:45:05.0656045Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2023-01-11T20:45:05.0719003Z Successfully installed ninja-1.10.2
2023-01-11T20:45:05.1335432Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T20:45:05.1336003Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T20:45:05.1336478Z + [[ linux-focal-py3.7-gcc7 == *asan* ]]
2023-01-11T20:45:05.1336762Z + [[ linux-focal-py3.7-gcc7 == *-tsan* ]]
2023-01-11T20:45:05.1355536Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2023-01-11T20:45:05.1355982Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2023-01-11T20:45:05.1356467Z + [[ linux-focal-py3.7-gcc7 == *tbb* ]]
2023-01-11T20:45:05.1356889Z + [[ linux-focal-py3.7-gcc7 == *libtorch* ]]
2023-01-11T20:45:05.1357166Z + [[ linux-focal-py3.7-gcc7 == *-bazel-* ]]
2023-01-11T20:45:05.1357442Z + [[ linux-focal-py3.7-gcc7 == *-tsan* ]]
2023-01-11T20:45:05.1357640Z + cd test
2023-01-11T20:45:05.1357895Z + python -c 'import torch; print(torch.__config__.show())'
2023-01-11T20:45:06.2178205Z PyTorch built with:
2023-01-11T20:45:06.2178749Z   - GCC 7.5
2023-01-11T20:45:06.2179121Z   - C++ Version: 201703
2023-01-11T20:45:06.2179860Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2023-01-11T20:45:06.2180627Z   - Intel(R) MKL-DNN v2.7.2 (Git Hash fbec3e25a559ee252022ae066817b204e106a6ba)
2023-01-11T20:45:06.2181170Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2023-01-11T20:45:06.2181673Z   - LAPACK is enabled (usually provided by MKL)
2023-01-11T20:45:06.2182122Z   - NNPACK is enabled
2023-01-11T20:45:06.2182545Z   - CPU capability usage: AVX2
2023-01-11T20:45:06.2187350Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2023-01-11T20:45:06.2190610Z 
2023-01-11T20:45:06.4083978Z + cd test
2023-01-11T20:45:06.4084485Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2023-01-11T20:45:07.4652326Z ATen/Parallel:
2023-01-11T20:45:07.4665071Z 	at::get_num_threads() : 4
2023-01-11T20:45:07.4665338Z 	at::get_num_interop_threads() : 4
2023-01-11T20:45:07.4665582Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2023-01-11T20:45:07.4665773Z 	omp_get_max_threads() : 4
2023-01-11T20:45:07.4666300Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2023-01-11T20:45:07.4666591Z 	mkl_get_max_threads() : 4
2023-01-11T20:45:07.4666909Z Intel(R) MKL-DNN v2.7.2 (Git Hash fbec3e25a559ee252022ae066817b204e106a6ba)
2023-01-11T20:45:07.4667179Z std::thread::hardware_concurrency() : 8
2023-01-11T20:45:07.4667393Z Environment variables:
2023-01-11T20:45:07.4667577Z 	OMP_NUM_THREADS : [not set]
2023-01-11T20:45:07.4667780Z 	MKL_NUM_THREADS : [not set]
2023-01-11T20:45:07.4667983Z ATen parallel backend: OpenMP
2023-01-11T20:45:07.4668111Z 
2023-01-11T20:45:07.6574646Z + [[ distributed == *backward* ]]
2023-01-11T20:45:07.6574986Z + [[ distributed == *xla* ]]
2023-01-11T20:45:07.6575419Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2023-01-11T20:45:07.6575991Z + [[ linux-focal-py3.7-gcc7 == *libtorch* ]]
2023-01-11T20:45:07.6576209Z + [[ distributed == distributed ]]
2023-01-11T20:45:07.6576404Z + install_filelock
2023-01-11T20:45:07.6576586Z + pip_install filelock
2023-01-11T20:45:07.6576827Z + pip install --progress-bar off filelock
2023-01-11T20:45:08.0291204Z Collecting filelock
2023-01-11T20:45:08.0454263Z   Downloading filelock-3.9.0-py3-none-any.whl (9.7 kB)
2023-01-11T20:45:08.7415025Z Installing collected packages: filelock
2023-01-11T20:45:08.7680525Z Successfully installed filelock-3.9.0
2023-01-11T20:45:08.8303565Z + install_triton
2023-01-11T20:45:08.8303916Z + local commit
2023-01-11T20:45:08.8304247Z + [[ distributed == *rocm* ]]
2023-01-11T20:45:08.8305637Z ++ get_pinned_commit triton
2023-01-11T20:45:08.8306016Z ++ cat .github/ci_commit_pins/triton.txt
2023-01-11T20:45:08.8317350Z + commit=0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T20:45:08.8318020Z + pip_install --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2023-01-11T20:45:08.8318558Z + pip install --progress-bar off --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2023-01-11T20:45:09.1615145Z Collecting git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2023-01-11T20:45:09.1621383Z   Cloning https://github.com/openai/triton (to revision 0d7e7532279e45672555e344646f5c19c3972331) to /tmp/pip-req-build-7hocamu9
2023-01-11T20:45:09.1651992Z   Running command git clone --filter=blob:none --quiet https://github.com/openai/triton /tmp/pip-req-build-7hocamu9
2023-01-11T20:45:10.0497695Z   Running command git rev-parse -q --verify 'sha^0d7e7532279e45672555e344646f5c19c3972331'
2023-01-11T20:45:10.0530395Z   Running command git fetch -q https://github.com/openai/triton 0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T20:45:10.3804864Z   Running command git checkout -q 0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T20:45:10.7383668Z   Resolved https://github.com/openai/triton to commit 0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T20:45:10.7384468Z   Running command git submodule update --init --recursive -q
2023-01-11T20:45:11.2011180Z   Preparing metadata (setup.py) ... [?25l- done
2023-01-11T20:45:11.3830276Z [?25hCollecting cmake
2023-01-11T20:45:11.3998586Z   Downloading cmake-3.25.0-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.7 MB)
2023-01-11T20:45:11.6788355Z Requirement already satisfied: filelock in /opt/conda/lib/python3.7/site-packages (from triton==2.0.0) (3.9.0)
2023-01-11T20:45:11.6792530Z Requirement already satisfied: torch in /opt/conda/lib/python3.7/site-packages (from triton==2.0.0) (2.0.0a0+git8419ddd)
2023-01-11T20:45:11.6996827Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.7/site-packages (from torch->triton==2.0.0) (4.4.0)
2023-01-11T20:45:11.7000923Z Requirement already satisfied: networkx in /opt/conda/lib/python3.7/site-packages (from torch->triton==2.0.0) (2.6.3)
2023-01-11T20:45:11.7005167Z Requirement already satisfied: sympy in /opt/conda/lib/python3.7/site-packages (from torch->triton==2.0.0) (1.10.1)
2023-01-11T20:45:11.7172181Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.7/site-packages (from sympy->torch->triton==2.0.0) (1.2.1)
2023-01-11T20:45:11.7231679Z Building wheels for collected packages: triton
2023-01-11T20:45:55.1808565Z   Building wheel for triton (setup.py) ... [?25l- \ | / - \ | / - \ | / - done
2023-01-11T20:45:55.2178322Z [?25h  Created wheel for triton: filename=triton-2.0.0-cp37-cp37m-linux_x86_64.whl size=15370740 sha256=6c92d679561e84c3c8adf4cabec2eb4afcdbcd56632e8baa5691b67f15826ff0
2023-01-11T20:45:55.2179532Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/aa/c5/3a/263a5ba27ed44796dab5f19b43135afb68beabac5d137652f7
2023-01-11T20:45:55.2203995Z Successfully built triton
2023-01-11T20:45:55.9126601Z Installing collected packages: cmake, triton
2023-01-11T20:45:57.0735455Z Successfully installed cmake-3.25.0 triton-2.0.0
2023-01-11T20:45:57.1747004Z + pip_install --user jinja2
2023-01-11T20:45:57.1747338Z + pip install --progress-bar off --user jinja2
2023-01-11T20:45:57.5510234Z Collecting jinja2
2023-01-11T20:45:57.5685906Z   Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
2023-01-11T20:45:57.5806203Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.7/site-packages (from jinja2) (2.1.1)
2023-01-11T20:45:58.2840647Z Installing collected packages: jinja2
2023-01-11T20:45:58.3576013Z Successfully installed jinja2-3.1.2
2023-01-11T20:45:58.4227788Z + test_distributed
2023-01-11T20:45:58.4228200Z + echo 'Testing distributed python tests'
2023-01-11T20:45:58.4228447Z Testing distributed python tests
2023-01-11T20:45:58.4228797Z + python test/run_test.py --distributed-tests --shard 2 2 --verbose
2023-01-11T20:45:59.8782061Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
2023-01-11T20:45:59.9998653Z Ignoring disabled issues:  ['91003']
2023-01-11T20:46:00.0171456Z Found test time stats from artifacts
2023-01-11T20:46:00.0185579Z Selected tests:
2023-01-11T20:46:00.0185948Z  distributed/algorithms/quantization/test_quantization
2023-01-11T20:46:00.0186428Z  distributed/test_distributed_spawn
2023-01-11T20:46:00.0186789Z  distributed/pipeline/sync/test_worker
2023-01-11T20:46:00.0187121Z  distributed/pipeline/sync/test_stream
2023-01-11T20:46:00.0187475Z  distributed/pipeline/sync/test_pipe
2023-01-11T20:46:00.0187784Z  distributed/pipeline/sync/test_microbatch
2023-01-11T20:46:00.0188129Z  distributed/pipeline/sync/test_dependency
2023-01-11T20:46:00.0188503Z  distributed/pipeline/sync/test_copy
2023-01-11T20:46:00.0188916Z  distributed/pipeline/sync/test_bugs
2023-01-11T20:46:00.0189265Z  distributed/pipeline/sync/skip/test_verify_skippables
2023-01-11T20:46:00.0195746Z  distributed/pipeline/sync/skip/test_stash_pop
2023-01-11T20:46:00.0196205Z  distributed/pipeline/sync/skip/test_leak
2023-01-11T20:46:00.0196568Z  distributed/pipeline/sync/skip/test_gpipe
2023-01-11T20:46:00.0196934Z  distributed/optim/test_named_optimizer
2023-01-11T20:46:00.0197342Z  distributed/fsdp/test_shard_utils
2023-01-11T20:46:00.0197736Z  distributed/elastic/events/lib_test
2023-01-11T20:46:00.0198129Z  distributed/_shard/test_sharder
2023-01-11T20:46:00.0198537Z  distributed/_shard/sharded_tensor/ops/test_math_ops
2023-01-11T20:46:00.0198962Z  distributed/test_c10d_spawn_ucc
2023-01-11T20:46:00.0199294Z  distributed/_shard/sharded_tensor/ops/test_softmax
2023-01-11T20:46:00.0199556Z  distributed/_shard/sharded_tensor/ops/test_embedding_bag
2023-01-11T20:46:00.0199811Z  distributed/_shard/sharded_tensor/ops/test_binary_cmp
2023-01-11T20:46:00.0200160Z  distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2023-01-11T20:46:00.0200484Z  distributed/_tools/test_memory_tracker
2023-01-11T20:46:00.0200735Z  distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2023-01-11T20:46:00.0201152Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2023-01-11T20:46:00.0201612Z  distributed/_shard/sharded_tensor/ops/test_embedding
2023-01-11T20:46:00.0202037Z  distributed/_shard/sharded_tensor/ops/test_chunk
2023-01-11T20:46:00.0202510Z  distributed/_shard/sharded_optim/test_sharded_optim
2023-01-11T20:46:00.0202814Z  distributed/test_c10d_spawn_nccl
2023-01-11T20:46:00.0203087Z  distributed/_shard/sharded_tensor/ops/test_init
2023-01-11T20:46:00.0203350Z  distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2023-01-11T20:46:00.0203652Z  distributed/_shard/test_partial_tensor
2023-01-11T20:46:00.0203896Z  distributed/_shard/sharded_tensor/ops/test_tensor_ops
2023-01-11T20:46:00.0204135Z  distributed/_shard/sharded_tensor/ops/test_linear
2023-01-11T20:46:00.0204440Z  distributed/_shard/sharding_plan/test_sharding_plan
2023-01-11T20:46:00.0204703Z  distributed/_shard/sharded_tensor/ops/test_matrix_ops
2023-01-11T20:46:00.0204939Z  distributed/_composable/test_checkpoint
2023-01-11T20:46:00.0205211Z  distributed/test_data_parallel
2023-01-11T20:46:00.0205425Z  distributed/nn/jit/test_instantiator
2023-01-11T20:46:00.0205619Z  distributed/test_launcher
2023-01-11T20:46:00.0205870Z  distributed/elastic/metrics/api_test
2023-01-11T20:46:00.0206098Z  distributed/checkpoint/test_planner
2023-01-11T20:46:00.0206309Z  distributed/_composable/test_contract
2023-01-11T20:46:00.0206544Z  distributed/elastic/utils/logging_test
2023-01-11T20:46:00.0206807Z  distributed/_tensor/test_pointwise_ops
2023-01-11T20:46:00.0207016Z  distributed/elastic/utils/util_test
2023-01-11T20:46:00.0207241Z  distributed/fsdp/test_checkpoint_wrapper
2023-01-11T20:46:00.0207515Z  distributed/test_multi_threaded_pg
2023-01-11T20:46:00.0207732Z  distributed/test_dynamo_distributed
2023-01-11T20:46:00.0207937Z  distributed/fsdp/test_fsdp_traversal
2023-01-11T20:46:00.0208238Z  distributed/tensor/parallel/test_view_sharding_dim_change
2023-01-11T20:46:00.0208599Z  distributed/rpc/test_share_memory
2023-01-11T20:46:00.0208818Z  distributed/fsdp/test_fsdp_multiple_forward
2023-01-11T20:46:00.0209104Z  distributed/elastic/utils/distributed_test
2023-01-11T20:46:00.0209357Z  distributed/fsdp/test_distributed_checkpoint
2023-01-11T20:46:00.0209578Z  distributed/checkpoint/test_checkpoint
2023-01-11T20:46:00.0209805Z  distributed/fsdp/test_fsdp_pure_fp16
2023-01-11T20:46:00.0210044Z  distributed/checkpoint/test_fsdp_model_state
2023-01-11T20:46:00.0210264Z  distributed/fsdp/test_fsdp_clip_grad_norm
2023-01-11T20:46:00.0210543Z  distributed/tensor/parallel/test_tp_examples
2023-01-11T20:46:00.0210777Z  distributed/fsdp/test_fsdp_hybrid_shard
2023-01-11T20:46:00.0211176Z  distributed/tensor/parallel/test_2d_parallel
2023-01-11T20:46:00.0211397Z  distributed/_tensor/test_api
2023-01-11T20:46:00.0211694Z  distributed/elastic/timer/local_timer_example
2023-01-11T20:46:00.0212048Z  distributed/fsdp/test_fsdp_ignored_modules
2023-01-11T20:46:00.0212499Z  distributed/_tensor/test_matrix_ops
2023-01-11T20:46:00.0212813Z  distributed/fsdp/test_fsdp_meta
2023-01-11T20:46:00.0213095Z  distributed/fsdp/test_fsdp_comm
2023-01-11T20:46:00.0213395Z  distributed/fsdp/test_fsdp_freezing_weights
2023-01-11T20:46:00.0213715Z  distributed/algorithms/test_join
2023-01-11T20:46:00.0214067Z  distributed/tensor/parallel/test_tp_style
2023-01-11T20:46:00.0214420Z  distributed/_tensor/test_redistribute
2023-01-11T20:46:00.0214754Z  distributed/test_store
2023-01-11T20:46:00.0215084Z  distributed/_composable/test_fully_shard
2023-01-11T20:46:00.0215426Z  distributed/fsdp/test_fsdp_grad_acc
2023-01-11T20:46:00.0215827Z  distributed/_tensor/test_view_ops
2023-01-11T20:46:00.0216171Z  distributed/fsdp/test_fsdp_checkpoint
2023-01-11T20:46:00.0216531Z  distributed/fsdp/test_fsdp_use_orig_params
2023-01-11T20:46:00.0216879Z  distributed/_tensor/test_tensor_ops
2023-01-11T20:46:00.0217214Z  distributed/fsdp/test_wrap
2023-01-11T20:46:00.0217526Z  distributed/test_c10d_nccl
2023-01-11T20:46:00.0217911Z  distributed/fsdp/test_fsdp_summon_full_params
2023-01-11T20:46:00.0218296Z  distributed/fsdp/test_fsdp_core
2023-01-11T20:46:00.0218638Z  distributed/fsdp/test_fsdp_mixed_precision
2023-01-11T20:46:00.0218963Z  distributed/fsdp/test_fsdp_optim_state
2023-01-11T20:46:00.0219289Z  distributed/fsdp/test_fsdp_state_dict
2023-01-11T20:46:00.0219614Z  distributed/rpc/test_faulty_agent
2023-01-11T20:46:00.0219934Z  distributed/rpc/cuda/test_tensorpipe_agent
2023-01-11T20:46:00.0220250Z  distributed/test_c10d_gloo
2023-01-11T20:46:00.0410731Z Prioritized test from test file changes.
2023-01-11T20:46:00.0411276Z reordering tests for PR:
2023-01-11T20:46:00.0412011Z prioritized: ['distributed/optim/test_named_optimizer', 'distributed/fsdp/test_fsdp_ignored_modules', 'distributed/fsdp/test_fsdp_optim_state']
2023-01-11T20:46:00.0418438Z the rest: ['distributed/algorithms/quantization/test_quantization', 'distributed/test_distributed_spawn', 'distributed/pipeline/sync/test_worker', 'distributed/pipeline/sync/test_stream', 'distributed/pipeline/sync/test_pipe', 'distributed/pipeline/sync/test_microbatch', 'distributed/pipeline/sync/test_dependency', 'distributed/pipeline/sync/test_copy', 'distributed/pipeline/sync/test_bugs', 'distributed/pipeline/sync/skip/test_verify_skippables', 'distributed/pipeline/sync/skip/test_stash_pop', 'distributed/pipeline/sync/skip/test_leak', 'distributed/pipeline/sync/skip/test_gpipe', 'distributed/fsdp/test_shard_utils', 'distributed/elastic/events/lib_test', 'distributed/_shard/test_sharder', 'distributed/_shard/sharded_tensor/ops/test_math_ops', 'distributed/test_c10d_spawn_ucc', 'distributed/_shard/sharded_tensor/ops/test_softmax', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks', 'distributed/_tools/test_memory_tracker', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard', 'distributed/_shard/sharded_tensor/test_megatron_prototype', 'distributed/_shard/sharded_tensor/ops/test_embedding', 'distributed/_shard/sharded_tensor/ops/test_chunk', 'distributed/_shard/sharded_optim/test_sharded_optim', 'distributed/test_c10d_spawn_nccl', 'distributed/_shard/sharded_tensor/ops/test_init', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops', 'distributed/_shard/test_partial_tensor', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops', 'distributed/_shard/sharded_tensor/ops/test_linear', 'distributed/_shard/sharding_plan/test_sharding_plan', 'distributed/_shard/sharded_tensor/ops/test_matrix_ops', 'distributed/_composable/test_checkpoint', 'distributed/test_data_parallel', 'distributed/nn/jit/test_instantiator', 'distributed/test_launcher', 'distributed/elastic/metrics/api_test', 'distributed/checkpoint/test_planner', 'distributed/_composable/test_contract', 'distributed/elastic/utils/logging_test', 'distributed/_tensor/test_pointwise_ops', 'distributed/elastic/utils/util_test', 'distributed/fsdp/test_checkpoint_wrapper', 'distributed/test_multi_threaded_pg', 'distributed/test_dynamo_distributed', 'distributed/fsdp/test_fsdp_traversal', 'distributed/tensor/parallel/test_view_sharding_dim_change', 'distributed/rpc/test_share_memory', 'distributed/fsdp/test_fsdp_multiple_forward', 'distributed/elastic/utils/distributed_test', 'distributed/fsdp/test_distributed_checkpoint', 'distributed/checkpoint/test_checkpoint', 'distributed/fsdp/test_fsdp_pure_fp16', 'distributed/checkpoint/test_fsdp_model_state', 'distributed/fsdp/test_fsdp_clip_grad_norm', 'distributed/tensor/parallel/test_tp_examples', 'distributed/fsdp/test_fsdp_hybrid_shard', 'distributed/tensor/parallel/test_2d_parallel', 'distributed/_tensor/test_api', 'distributed/elastic/timer/local_timer_example', 'distributed/_tensor/test_matrix_ops', 'distributed/fsdp/test_fsdp_meta', 'distributed/fsdp/test_fsdp_comm', 'distributed/fsdp/test_fsdp_freezing_weights', 'distributed/algorithms/test_join', 'distributed/tensor/parallel/test_tp_style', 'distributed/_tensor/test_redistribute', 'distributed/test_store', 'distributed/_composable/test_fully_shard', 'distributed/fsdp/test_fsdp_grad_acc', 'distributed/_tensor/test_view_ops', 'distributed/fsdp/test_fsdp_checkpoint', 'distributed/fsdp/test_fsdp_use_orig_params', 'distributed/_tensor/test_tensor_ops', 'distributed/fsdp/test_wrap', 'distributed/test_c10d_nccl', 'distributed/fsdp/test_fsdp_summon_full_params', 'distributed/fsdp/test_fsdp_core', 'distributed/fsdp/test_fsdp_mixed_precision', 'distributed/fsdp/test_fsdp_state_dict', 'distributed/rpc/test_faulty_agent', 'distributed/rpc/cuda/test_tensorpipe_agent', 'distributed/test_c10d_gloo']
2023-01-11T20:46:00.0422472Z 
2023-01-11T20:46:00.0422889Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2023-01-11T20:46:00.0606964Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2023-01-11T20:46:00.0742220Z parallel (file granularity) tests:
2023-01-11T20:46:00.0742534Z  
2023-01-11T20:46:00.0742827Z serial (file granularity) tests:
2023-01-11T20:46:00.0743060Z  distributed/optim/test_named_optimizer
2023-01-11T20:46:00.0743297Z  distributed/fsdp/test_fsdp_ignored_modules
2023-01-11T20:46:00.0743529Z  distributed/fsdp/test_fsdp_optim_state
2023-01-11T20:46:00.0743801Z  distributed/algorithms/quantization/test_quantization
2023-01-11T20:46:00.0744199Z  distributed/test_distributed_spawn
2023-01-11T20:46:00.0744555Z  distributed/pipeline/sync/test_worker
2023-01-11T20:46:00.0744964Z  distributed/pipeline/sync/test_stream
2023-01-11T20:46:00.0745364Z  distributed/pipeline/sync/test_pipe
2023-01-11T20:46:00.0745782Z  distributed/pipeline/sync/test_microbatch
2023-01-11T20:46:00.0746192Z  distributed/pipeline/sync/test_dependency
2023-01-11T20:46:00.0746647Z  distributed/pipeline/sync/test_copy
2023-01-11T20:46:00.0747055Z  distributed/pipeline/sync/test_bugs
2023-01-11T20:46:00.0747533Z  distributed/pipeline/sync/skip/test_verify_skippables
2023-01-11T20:46:00.0747807Z  distributed/pipeline/sync/skip/test_stash_pop
2023-01-11T20:46:00.0748172Z  distributed/pipeline/sync/skip/test_leak
2023-01-11T20:46:00.0748553Z  distributed/pipeline/sync/skip/test_gpipe
2023-01-11T20:46:00.0748947Z  distributed/fsdp/test_shard_utils
2023-01-11T20:46:00.0749344Z  distributed/elastic/events/lib_test
2023-01-11T20:46:00.0749737Z  distributed/_shard/test_sharder
2023-01-11T20:46:00.0750146Z  distributed/_shard/sharded_tensor/ops/test_math_ops
2023-01-11T20:46:00.0750569Z  distributed/test_c10d_spawn_ucc
2023-01-11T20:46:00.0750977Z  distributed/_shard/sharded_tensor/ops/test_softmax
2023-01-11T20:46:00.0751449Z  distributed/_shard/sharded_tensor/ops/test_embedding_bag
2023-01-11T20:46:00.0751728Z  distributed/_shard/sharded_tensor/ops/test_binary_cmp
2023-01-11T20:46:00.0751996Z  distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2023-01-11T20:46:00.0752307Z  distributed/_tools/test_memory_tracker
2023-01-11T20:46:00.0752553Z  distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2023-01-11T20:46:00.0752830Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2023-01-11T20:46:00.0753092Z  distributed/_shard/sharded_tensor/ops/test_embedding
2023-01-11T20:46:00.0753331Z  distributed/_shard/sharded_tensor/ops/test_chunk
2023-01-11T20:46:00.0753584Z  distributed/_shard/sharded_optim/test_sharded_optim
2023-01-11T20:46:00.0753815Z  distributed/test_c10d_spawn_nccl
2023-01-11T20:46:00.0754034Z  distributed/_shard/sharded_tensor/ops/test_init
2023-01-11T20:46:00.0754295Z  distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2023-01-11T20:46:00.0754541Z  distributed/_shard/test_partial_tensor
2023-01-11T20:46:00.0754769Z  distributed/_shard/sharded_tensor/ops/test_tensor_ops
2023-01-11T20:46:00.0755020Z  distributed/_shard/sharded_tensor/ops/test_linear
2023-01-11T20:46:00.0755272Z  distributed/_shard/sharding_plan/test_sharding_plan
2023-01-11T20:46:00.0755530Z  distributed/_shard/sharded_tensor/ops/test_matrix_ops
2023-01-11T20:46:00.0755760Z  distributed/_composable/test_checkpoint
2023-01-11T20:46:00.0755976Z  distributed/test_data_parallel
2023-01-11T20:46:00.0756191Z  distributed/nn/jit/test_instantiator
2023-01-11T20:46:00.0756384Z  distributed/test_launcher
2023-01-11T20:46:00.0756591Z  distributed/elastic/metrics/api_test
2023-01-11T20:46:00.0756811Z  distributed/checkpoint/test_planner
2023-01-11T20:46:00.0757021Z  distributed/_composable/test_contract
2023-01-11T20:46:00.0757242Z  distributed/elastic/utils/logging_test
2023-01-11T20:46:00.0757466Z  distributed/_tensor/test_pointwise_ops
2023-01-11T20:46:00.0757673Z  distributed/elastic/utils/util_test
2023-01-11T20:46:00.0757898Z  distributed/fsdp/test_checkpoint_wrapper
2023-01-11T20:46:00.0758122Z  distributed/test_multi_threaded_pg
2023-01-11T20:46:00.0758325Z  distributed/test_dynamo_distributed
2023-01-11T20:46:00.0758543Z  distributed/fsdp/test_fsdp_traversal
2023-01-11T20:46:00.0758793Z  distributed/tensor/parallel/test_view_sharding_dim_change
2023-01-11T20:46:00.0759022Z  distributed/rpc/test_share_memory
2023-01-11T20:46:00.0759249Z  distributed/fsdp/test_fsdp_multiple_forward
2023-01-11T20:46:00.0759487Z  distributed/elastic/utils/distributed_test
2023-01-11T20:46:00.0759726Z  distributed/fsdp/test_distributed_checkpoint
2023-01-11T20:46:00.0759946Z  distributed/checkpoint/test_checkpoint
2023-01-11T20:46:00.0760167Z  distributed/fsdp/test_fsdp_pure_fp16
2023-01-11T20:46:00.0760397Z  distributed/checkpoint/test_fsdp_model_state
2023-01-11T20:46:00.0760621Z  distributed/fsdp/test_fsdp_clip_grad_norm
2023-01-11T20:46:00.0760856Z  distributed/tensor/parallel/test_tp_examples
2023-01-11T20:46:00.0761106Z  distributed/fsdp/test_fsdp_hybrid_shard
2023-01-11T20:46:00.0761330Z  distributed/tensor/parallel/test_2d_parallel
2023-01-11T20:46:00.0761546Z  distributed/_tensor/test_api
2023-01-11T20:46:00.0761768Z  distributed/elastic/timer/local_timer_example
2023-01-11T20:46:00.0761985Z  distributed/_tensor/test_matrix_ops
2023-01-11T20:46:00.0762255Z  distributed/fsdp/test_fsdp_meta
2023-01-11T20:46:00.0762462Z  distributed/fsdp/test_fsdp_comm
2023-01-11T20:46:00.0762692Z  distributed/fsdp/test_fsdp_freezing_weights
2023-01-11T20:46:00.0762903Z  distributed/algorithms/test_join
2023-01-11T20:46:00.0763127Z  distributed/tensor/parallel/test_tp_style
2023-01-11T20:46:00.0763354Z  distributed/_tensor/test_redistribute
2023-01-11T20:46:00.0763546Z  distributed/test_store
2023-01-11T20:46:00.0763754Z  distributed/_composable/test_fully_shard
2023-01-11T20:46:00.0763977Z  distributed/fsdp/test_fsdp_grad_acc
2023-01-11T20:46:00.0764178Z  distributed/_tensor/test_view_ops
2023-01-11T20:46:00.0764398Z  distributed/fsdp/test_fsdp_checkpoint
2023-01-11T20:46:00.0764626Z  distributed/fsdp/test_fsdp_use_orig_params
2023-01-11T20:46:00.0764839Z  distributed/_tensor/test_tensor_ops
2023-01-11T20:46:00.0765043Z  distributed/fsdp/test_wrap
2023-01-11T20:46:00.0765240Z  distributed/test_c10d_nccl
2023-01-11T20:46:00.0765489Z  distributed/fsdp/test_fsdp_summon_full_params
2023-01-11T20:46:00.0765716Z  distributed/fsdp/test_fsdp_core
2023-01-11T20:46:00.0765940Z  distributed/fsdp/test_fsdp_mixed_precision
2023-01-11T20:46:00.0766153Z  distributed/fsdp/test_fsdp_state_dict
2023-01-11T20:46:00.0766368Z  distributed/rpc/test_faulty_agent
2023-01-11T20:46:00.0766599Z  distributed/rpc/cuda/test_tensorpipe_agent
2023-01-11T20:46:00.0766797Z  distributed/test_c10d_gloo
2023-01-11T20:46:01.5608864Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
2023-01-11T20:46:01.5713192Z No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
2023-01-11T20:46:01.6930468Z Ignoring disabled issues:  ['91003']
2023-01-11T20:46:01.6987310Z Ignoring disabled issues:  ['91003']
2023-01-11T20:46:02.0495051Z Running distributed/optim/test_named_optimizer ... [2023-01-11 20:46:02.049120]
2023-01-11T20:46:02.0497238Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_named_optimizer.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:46:02.049493]
2023-01-11T20:46:03.5025933Z 
2023-01-11T20:46:03.5026528Z Expand the folded group to see the log file of distributed/optim/test_named_optimizer
2023-01-11T20:46:03.5031951Z ##[group]PRINTING LOG FILE of distributed/optim/test_named_optimizer (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_named_optimizer_6vylopsy)
2023-01-11T20:46:03.5032269Z 
2023-01-11T20:46:03.5032526Z ##[endgroup]
2023-01-11T20:46:03.5033288Z FINISHED PRINTING LOG FILE of distributed/optim/test_named_optimizer (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_named_optimizer_6vylopsy)
2023-01-11T20:46:03.5033667Z 
2023-01-11T20:46:03.5034091Z Running distributed/fsdp/test_fsdp_ignored_modules ... [2023-01-11 20:46:03.503195]
2023-01-11T20:46:03.5037670Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_ignored_modules.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:46:03.503531]
2023-01-11T20:46:16.4565853Z 
2023-01-11T20:46:16.4567034Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_ignored_modules
2023-01-11T20:46:16.4568193Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_ignored_modules (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_ignored_modules_6o4i4uai)
2023-01-11T20:46:16.4569287Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules
2023-01-11T20:46:16.4569659Z 
2023-01-11T20:46:16.4569761Z Running tests...
2023-01-11T20:46:16.4570220Z ----------------------------------------------------------------------
2023-01-11T20:46:16.4570696Z   test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_False (__main__.TestFSDPIgnoredModules)
2023-01-11T20:46:16.4571454Z Tests ignoring different modules across ranks. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 711
2023-01-11T20:46:16.4571947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 712
2023-01-11T20:46:16.4572679Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 713
2023-01-11T20:46:16.4573121Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 714
2023-01-11T20:46:16.4573770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4574275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4574926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4575426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4576016Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:46:16.4576799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4577530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4578196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4578702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4579216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:46:16.4580026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4580563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4581326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4581834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4582324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:46:16.4583103Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4583902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4584401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4584893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4585225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4585560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:46:16.4586056Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4586564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4587078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4587462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:46:16.4587804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:46:16.4588129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:46:16.4588458Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:46:16.4588714Z dist init r=0, world=4
2023-01-11T20:46:16.4588879Z dist init r=1, world=4
2023-01-11T20:46:16.4589052Z dist init r=3, world=4
2023-01-11T20:46:16.4589228Z dist init r=2, world=4
2023-01-11T20:46:16.4589419Z skip: Need at least 2 CUDA devices (2.652s)
2023-01-11T20:46:16.4589849Z   test_diff_ignored_modules_across_ranks_pass_ignored_modules_to_root_True (__main__.TestFSDPIgnoredModules)
2023-01-11T20:46:16.4590258Z Tests ignoring different modules across ranks. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 767
2023-01-11T20:46:16.4590623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 768
2023-01-11T20:46:16.4590927Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 769
2023-01-11T20:46:16.4591240Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 770
2023-01-11T20:46:16.4591684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4592010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4592419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4592812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4593146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:46:16.4593597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4593918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4594337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4594676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4594990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:46:16.4595454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4595775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4596186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4596523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4596853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:46:16.4597341Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4597819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4598141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4598563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4598903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4599223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:46:16.4599712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4600223Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4600730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4601099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:46:16.4601441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:46:16.4601778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:46:16.4602109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:46:16.4602401Z dist init r=3, world=4
2023-01-11T20:46:16.4602579Z dist init r=2, world=4
2023-01-11T20:46:16.4602740Z dist init r=0, world=4
2023-01-11T20:46:16.4602915Z dist init r=1, world=4
2023-01-11T20:46:16.4603120Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T20:46:16.4603384Z   test_ignored_modules_invalid (__main__.TestFSDPIgnoredModules)
2023-01-11T20:46:16.4603756Z Tests that passing an FSDP module as an ignored module or the ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 823
2023-01-11T20:46:16.4604140Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 824
2023-01-11T20:46:16.4604461Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 825
2023-01-11T20:46:16.4604768Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 826
2023-01-11T20:46:16.4605241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4605567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4605990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4606320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4606650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:46:16.4607114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4607421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4607840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4608179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4608510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:46:16.4608954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4609274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4609691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4610027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4610338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:46:16.4610797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4611274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4611687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4612022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4612351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:46:16.4612840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4613343Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4613851Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4614362Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4614736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:46:16.4615148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:46:16.4615482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:46:16.4615893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:46:16.4616138Z dist init r=0, world=4
2023-01-11T20:46:16.4616316Z dist init r=3, world=4
2023-01-11T20:46:16.4616490Z dist init r=2, world=4
2023-01-11T20:46:16.4616653Z dist init r=1, world=4
2023-01-11T20:46:16.4616935Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T20:46:16.4617198Z   test_ignored_modules_nested (__main__.TestFSDPIgnoredModules)
2023-01-11T20:46:16.4617582Z Tests that passing a module with nested FSDP modules does not ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 879
2023-01-11T20:46:16.4617968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 880
2023-01-11T20:46:16.4618348Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 881
2023-01-11T20:46:16.4618662Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 882
2023-01-11T20:46:16.4619114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4619427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4619851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4620191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4620524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:46:16.4620976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4621298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4621715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4622051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4622368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:46:16.4622826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4623143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4623549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4623886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4624212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:46:16.4624698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4625174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4625496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4625917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4626244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4626574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:46:16.4627058Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4627576Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4628113Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4628493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:46:16.4628834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:46:16.4629171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:46:16.4629495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:46:16.4629743Z dist init r=1, world=4
2023-01-11T20:46:16.4629921Z dist init r=3, world=4
2023-01-11T20:46:16.4630084Z dist init r=2, world=4
2023-01-11T20:46:16.4630255Z dist init r=0, world=4
2023-01-11T20:46:16.4630456Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T20:46:16.4630851Z   test_ignored_modules_not_under_wrapped_root (__main__.TestFSDPIgnoredModules) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 935
2023-01-11T20:46:16.4631262Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 936
2023-01-11T20:46:16.4631582Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 937
2023-01-11T20:46:16.4631896Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 938
2023-01-11T20:46:16.4632331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4632655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4633078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4633415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4633734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:46:16.4634197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4634517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4634923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4635262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4635589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:46:16.4636051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4636358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4636778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4637116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4637428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:46:16.4637886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4638207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4638624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4638947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4639274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:46:16.4639767Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4640317Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4640814Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4641321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4641703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:46:16.4642045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:46:16.4642370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:46:16.4642703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:46:16.4642956Z dist init r=1, world=4
2023-01-11T20:46:16.4643153Z dist init r=2, world=4
2023-01-11T20:46:16.4643330Z dist init r=3, world=4
2023-01-11T20:46:16.4643503Z dist init r=0, world=4
2023-01-11T20:46:16.4643696Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T20:46:16.4643970Z   test_ignored_modules_transformer (__main__.TestFSDPIgnoredModules)
2023-01-11T20:46:16.4644462Z Tests that ignored modules' parameters are not flattened for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 991
2023-01-11T20:46:16.4644846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 992
2023-01-11T20:46:16.4645154Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 993
2023-01-11T20:46:16.4645469Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 994
2023-01-11T20:46:16.4645912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4646228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4646656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4646993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4647321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:46:16.4647771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4648089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4648506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4648833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4649157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:46:16.4649619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4649938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4650343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4650683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4651217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:46:16.4651683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:46:16.4651988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:46:16.4652408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:46:16.4652816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:46:16.4653132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:46:16.4653619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4654132Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4654642Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4655132Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:46:16.4655511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:46:16.4655991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:46:16.4656337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:46:16.4656665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:46:16.4656918Z dist init r=0, world=4
2023-01-11T20:46:16.4657094Z dist init r=2, world=4
2023-01-11T20:46:16.4657254Z dist init r=3, world=4
2023-01-11T20:46:16.4657430Z dist init r=1, world=4
2023-01-11T20:46:16.4657634Z skip: Need at least 2 CUDA devices (1.819s)
2023-01-11T20:46:16.4657768Z 
2023-01-11T20:46:16.4657961Z ----------------------------------------------------------------------
2023-01-11T20:46:16.4658197Z Ran 6 tests in 11.347s
2023-01-11T20:46:16.4658308Z 
2023-01-11T20:46:16.4658379Z OK (skipped=6)
2023-01-11T20:46:16.4658482Z 
2023-01-11T20:46:16.4658564Z Generating XML reports...
2023-01-11T20:46:16.4659037Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20230111204604.xml
2023-01-11T20:46:16.4659321Z 
2023-01-11T20:46:16.4659653Z ##[endgroup]
2023-01-11T20:46:16.4660127Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_ignored_modules (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_ignored_modules_6o4i4uai)
2023-01-11T20:46:16.4660389Z 
2023-01-11T20:46:16.4660592Z Running distributed/fsdp/test_fsdp_optim_state ... [2023-01-11 20:46:16.457155]
2023-01-11T20:46:16.4661113Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_optim_state.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:46:16.457577]
2023-01-11T20:48:01.1459698Z 
2023-01-11T20:48:01.1460227Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_optim_state
2023-01-11T20:48:01.1461315Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_optim_state (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_optim_state_v5kxh7_d)
2023-01-11T20:48:01.1467975Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state
2023-01-11T20:48:01.1468328Z 
2023-01-11T20:48:01.1468406Z Running tests...
2023-01-11T20:48:01.1468877Z ----------------------------------------------------------------------
2023-01-11T20:48:01.1469442Z   test_compatible_with_named_optimizer (__main__.TestFSDPOptimState) ... skip: The test currently fails on CI. (0.002s)
2023-01-11T20:48:01.1469995Z   test_flatten_sharded_optim_state_dict_nested (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1470806Z Tests :meth:`flatten_sharded_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1058
2023-01-11T20:48:01.1471553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1059
2023-01-11T20:48:01.1472010Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1060
2023-01-11T20:48:01.1472447Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1061
2023-01-11T20:48:01.1493087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1493677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1494442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1495031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1495580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1496314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1496644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1497071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1515470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1516135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1517102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1517472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1517895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1518482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1518868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1519335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1519847Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1520556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1521104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1521582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1522506Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1523455Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1524317Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1525162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1525881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1526532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1527127Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1527492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1527746Z dist init r=1, world=4
2023-01-11T20:48:01.1527929Z dist init r=0, world=4
2023-01-11T20:48:01.1528093Z dist init r=3, world=4
2023-01-11T20:48:01.1528271Z dist init r=2, world=4
2023-01-11T20:48:01.1528613Z skip: Need at least 2 CUDA devices (2.275s)
2023-01-11T20:48:01.1529122Z   test_flatten_sharded_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1529872Z Tests :meth:`flatten_sharded_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1114
2023-01-11T20:48:01.1530387Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1115
2023-01-11T20:48:01.1530709Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1116
2023-01-11T20:48:01.1531173Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1117
2023-01-11T20:48:01.1531618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1531952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1532381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1532713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1533049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1533805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1534388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1535122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1535757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1536446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1537173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1537792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1538559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1539166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1539684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1540497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1541317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1541849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1542580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1543149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1543698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1544540Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1545395Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1546237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1546863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1547437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1547981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1548543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1548962Z dist init r=1, world=4
2023-01-11T20:48:01.1549244Z dist init r=0, world=4
2023-01-11T20:48:01.1549537Z dist init r=2, world=4
2023-01-11T20:48:01.1549959Z dist init r=3, world=4
2023-01-11T20:48:01.1550295Z skip: Need at least 2 CUDA devices (2.020s)
2023-01-11T20:48:01.1550748Z   test_full_optim_state_dict_keys (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1551353Z Tests that the parameter keys returned by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1170
2023-01-11T20:48:01.1552007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1171
2023-01-11T20:48:01.1552565Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1172
2023-01-11T20:48:01.1553135Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1173
2023-01-11T20:48:01.1553981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1554581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1555452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1556099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1556721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1557597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1558204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1558757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1559100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1559417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1559884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1560212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1560622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1560966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1561295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1561758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1562066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1562487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1562825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1563145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1563638Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1564157Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1564669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1565338Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1565773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1566385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1566861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1567256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1567510Z dist init r=2, world=4
2023-01-11T20:48:01.1567691Z dist init r=0, world=4
2023-01-11T20:48:01.1567859Z dist init r=1, world=4
2023-01-11T20:48:01.1568040Z dist init r=3, world=4
2023-01-11T20:48:01.1568245Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1568505Z   test_full_optim_state_dict_nested_invalid (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1568889Z Tests that :meth:`full_optim_state_dict` raises an error when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1226
2023-01-11T20:48:01.1569390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1227
2023-01-11T20:48:01.1569786Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1228
2023-01-11T20:48:01.1570094Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1229
2023-01-11T20:48:01.1570614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1571109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1571546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1571880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1572215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1572679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1572993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1573416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1573760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1574093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1574565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1575054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1575380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1575789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1576183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1576516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1576981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1577296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1577721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1578066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1579140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1579612Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1580126Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1580643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1581118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1581452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1581792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1582130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1582372Z dist init r=1, world=4
2023-01-11T20:48:01.1582551Z dist init r=0, world=4
2023-01-11T20:48:01.1582728Z dist init r=2, world=4
2023-01-11T20:48:01.1582889Z dist init r=3, world=4
2023-01-11T20:48:01.1583097Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1583352Z   test_optim_input_warning (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1583725Z Tests that passing the ``optim_input`` argument into optimizer state ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1282
2023-01-11T20:48:01.1584150Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1283
2023-01-11T20:48:01.1584663Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1284
2023-01-11T20:48:01.1585175Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1285
2023-01-11T20:48:01.1585856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1586186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1586620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1586964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1587288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1587753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1588083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1588495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1588831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1589162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1589623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1589936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1590356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1590696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1591028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1591476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1591804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1592221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1592545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1592877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1593360Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1593875Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1594438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1594934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1595315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1595660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1595989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1596328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1596583Z dist init r=2, world=4
2023-01-11T20:48:01.1596763Z dist init r=1, world=4
2023-01-11T20:48:01.1596928Z dist init r=3, world=4
2023-01-11T20:48:01.1597103Z dist init r=0, world=4
2023-01-11T20:48:01.1597346Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1597724Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1598219Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1338
2023-01-11T20:48:01.1598608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1339
2023-01-11T20:48:01.1598925Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1340
2023-01-11T20:48:01.1599230Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1341
2023-01-11T20:48:01.1599675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1599999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1600415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1600754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1601086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1601551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1601860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1602281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1602615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1602931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1603417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1603910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1604230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1604640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1604979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1605306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1605793Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1606273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1606629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1607053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1607397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1607885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1608490Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1609162Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1609533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1609880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1610265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1610607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1610982Z dist init r=0, world=4
2023-01-11T20:48:01.1611181Z dist init r=3, world=4
2023-01-11T20:48:01.1611360Z dist init r=2, world=4
2023-01-11T20:48:01.1611526Z dist init r=1, world=4
2023-01-11T20:48:01.1611734Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1612126Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1612746Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1394
2023-01-11T20:48:01.1613192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1395
2023-01-11T20:48:01.1613521Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1396
2023-01-11T20:48:01.1613839Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1397
2023-01-11T20:48:01.1614289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1614616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1615040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1615485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1615806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1616407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1616732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1617142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1617481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1617815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1618283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1618593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1619020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1619358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1619689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1620229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1620559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1620987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1621342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1621676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1622169Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1622688Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1623229Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1623748Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1624133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1624481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1624811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1625146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1625406Z dist init r=1, world=4
2023-01-11T20:48:01.1625574Z dist init r=3, world=4
2023-01-11T20:48:01.1625755Z dist init r=2, world=4
2023-01-11T20:48:01.1625934Z dist init r=0, world=4
2023-01-11T20:48:01.1626128Z skip: Need at least 2 CUDA devices (1.920s)
2023-01-11T20:48:01.1626522Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1627024Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1450
2023-01-11T20:48:01.1627416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1451
2023-01-11T20:48:01.1627726Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1452
2023-01-11T20:48:01.1628041Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1453
2023-01-11T20:48:01.1628494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1628824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1629371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1629935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1630465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1630999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1631322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1631745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1632089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1632407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1632872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1633254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1633680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1634006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1634337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1634800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1635111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1635529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1635868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1636236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1636716Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1637231Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1637745Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1638252Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1638618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1638961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1639298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1639631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1639894Z dist init r=1, world=4
2023-01-11T20:48:01.1640075Z dist init r=0, world=4
2023-01-11T20:48:01.1640239Z dist init r=3, world=4
2023-01-11T20:48:01.1640414Z dist init r=2, world=4
2023-01-11T20:48:01.1640616Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1641004Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1641488Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1506
2023-01-11T20:48:01.1641872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1507
2023-01-11T20:48:01.1642193Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1508
2023-01-11T20:48:01.1642515Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1509
2023-01-11T20:48:01.1642949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1643272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1643700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1644029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1644361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1644823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1645148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1645558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1645942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1646275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1646727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1647045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1647464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1647802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1648117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1648640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1649141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1649461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1649871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1650210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1650536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1651156Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1651676Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1652193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1652582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1652912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1653248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1653587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1653844Z dist init r=0, world=4
2023-01-11T20:48:01.1654009Z dist init r=2, world=4
2023-01-11T20:48:01.1654183Z dist init r=1, world=4
2023-01-11T20:48:01.1654357Z dist init r=3, world=4
2023-01-11T20:48:01.1654547Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1654936Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1655435Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1562
2023-01-11T20:48:01.1655823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1563
2023-01-11T20:48:01.1656192Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1564
2023-01-11T20:48:01.1656507Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1565
2023-01-11T20:48:01.1656960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1657272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1657701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1658043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1658446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1658900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1659227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1659651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1659977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1660311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1660769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1661093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1661561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1661901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1662230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1662717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1663197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1663521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1663943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1664271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1664606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1665098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1665614Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1666118Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1666498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1666840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1667178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1667507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1667760Z dist init r=1, world=4
2023-01-11T20:48:01.1667937Z dist init r=2, world=4
2023-01-11T20:48:01.1668098Z dist init r=0, world=4
2023-01-11T20:48:01.1668271Z dist init r=3, world=4
2023-01-11T20:48:01.1668471Z skip: Need at least 2 CUDA devices (1.929s)
2023-01-11T20:48:01.1668846Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1669336Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1618
2023-01-11T20:48:01.1669723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1619
2023-01-11T20:48:01.1670045Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1620
2023-01-11T20:48:01.1670349Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1621
2023-01-11T20:48:01.1670833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1671156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1671584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1671914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1672243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1672705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1673016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1673439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1673815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1674148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1674626Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1675112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1675434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1675845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1676192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1676524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1676992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1677307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1677730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1678073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1678406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1678882Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1679401Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1679914Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1680299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1680632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1680970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1681313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1681560Z dist init r=1, world=4
2023-01-11T20:48:01.1681743Z dist init r=0, world=4
2023-01-11T20:48:01.1681923Z dist init r=2, world=4
2023-01-11T20:48:01.1682083Z dist init r=3, world=4
2023-01-11T20:48:01.1682288Z skip: Need at least 2 CUDA devices (2.019s)
2023-01-11T20:48:01.1682674Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1683174Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1674
2023-01-11T20:48:01.1683586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1675
2023-01-11T20:48:01.1683909Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1676
2023-01-11T20:48:01.1684231Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1677
2023-01-11T20:48:01.1684666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1684998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1685424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1685763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1686123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1686594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1686917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1687338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1687662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1687990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1688450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1688759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1689182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1689520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1689849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1690294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1690615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1692215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1692545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1692873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1693365Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1693890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1694388Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1694897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1695278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1695620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1696005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1696342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1696602Z dist init r=1, world=4
2023-01-11T20:48:01.1696866Z dist init r=2, world=4
2023-01-11T20:48:01.1697042Z dist init r=3, world=4
2023-01-11T20:48:01.1697221Z dist init r=0, world=4
2023-01-11T20:48:01.1697413Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1697801Z   test_optim_state_dict_nested_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1698297Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1730
2023-01-11T20:48:01.1698691Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1731
2023-01-11T20:48:01.1699002Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1732
2023-01-11T20:48:01.1699318Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1733
2023-01-11T20:48:01.1699807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1700139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1700554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1700894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1701224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1701675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1701993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1702413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1702751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1703072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1703535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1703863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1704284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1704606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1704938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1705397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1705707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1706126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1706461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1706791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1707264Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1707781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1708295Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1708800Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1709167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1709543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1709885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1710211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1710465Z dist init r=3, world=4
2023-01-11T20:48:01.1710645Z dist init r=2, world=4
2023-01-11T20:48:01.1710820Z dist init r=1, world=4
2023-01-11T20:48:01.1710981Z dist init r=0, world=4
2023-01-11T20:48:01.1711185Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1711576Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1712086Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1786
2023-01-11T20:48:01.1712474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1787
2023-01-11T20:48:01.1712797Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1788
2023-01-11T20:48:01.1713116Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1789
2023-01-11T20:48:01.1713551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1713876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1714303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1714632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1714965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1715428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1715753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1716159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1716497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1716826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1717332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1717643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1718064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1718403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1718719Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1719181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1719502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1719922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1720247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1720577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1721068Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1721584Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1722119Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1722630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1723009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1723341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1723729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1724058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1724314Z dist init r=1, world=4
2023-01-11T20:48:01.1724478Z dist init r=0, world=4
2023-01-11T20:48:01.1724654Z dist init r=3, world=4
2023-01-11T20:48:01.1724863Z dist init r=2, world=4
2023-01-11T20:48:01.1725058Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1725447Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1725944Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1842
2023-01-11T20:48:01.1726332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1843
2023-01-11T20:48:01.1726641Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1844
2023-01-11T20:48:01.1726956Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1845
2023-01-11T20:48:01.1727401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1727717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1728142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1728483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1728812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1729262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1729584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1730004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1730328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1730662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1731292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1731618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1732027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1732370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1732703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1733195Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1733693Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1734189Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1734583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1735001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1735342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1735674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1736225Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1736725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1737112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1737503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1737851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1738183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1738437Z dist init r=2, world=4
2023-01-11T20:48:01.1738621Z dist init r=0, world=4
2023-01-11T20:48:01.1738788Z dist init r=3, world=4
2023-01-11T20:48:01.1738966Z dist init r=1, world=4
2023-01-11T20:48:01.1739175Z skip: Need at least 2 CUDA devices (1.920s)
2023-01-11T20:48:01.1739556Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1740051Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1898
2023-01-11T20:48:01.1740443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1899
2023-01-11T20:48:01.1740772Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1900
2023-01-11T20:48:01.1741079Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1901
2023-01-11T20:48:01.1741528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1741852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1742280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1742610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1742944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1743413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1743728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1744151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1744494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1744829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1745278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1745603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1746021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1746362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1746682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1747176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1747496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1747907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1748247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1748574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1749058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1749562Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1750118Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1750633Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1751014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1751341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1751674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1752021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1752289Z dist init r=2, world=4
2023-01-11T20:48:01.1752471Z dist init r=1, world=4
2023-01-11T20:48:01.1752650Z dist init r=0, world=4
2023-01-11T20:48:01.1752816Z dist init r=3, world=4
2023-01-11T20:48:01.1753021Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1753419Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1753918Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1954
2023-01-11T20:48:01.1754295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1955
2023-01-11T20:48:01.1754621Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1956
2023-01-11T20:48:01.1754940Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1957
2023-01-11T20:48:01.1755390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1755703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1756133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1756478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1756796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1757259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1757581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1758003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1758330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1758659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1759150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1759663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1759982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1760400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1760739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1761055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1761515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1761835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1762285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1762613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1762949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1763434Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1763936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1764457Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1764842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1765186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1765513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1765847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1766102Z dist init r=0, world=4
2023-01-11T20:48:01.1766283Z dist init r=2, world=4
2023-01-11T20:48:01.1766445Z dist init r=1, world=4
2023-01-11T20:48:01.1766620Z dist init r=3, world=4
2023-01-11T20:48:01.1766822Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1767200Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1767700Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2010
2023-01-11T20:48:01.1768092Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2011
2023-01-11T20:48:01.1768403Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2012
2023-01-11T20:48:01.1768724Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2013
2023-01-11T20:48:01.1769165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1769490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1769897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1770237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1770564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1771191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1771504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1771993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1772331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1772651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1773149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1773646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1773971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1774382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1774728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1775100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1775566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1775874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1776363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1776703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1777021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1777509Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1778030Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1778544Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1778916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1779262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1779602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1779942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1780186Z dist init r=0, world=4
2023-01-11T20:48:01.1780364Z dist init r=3, world=4
2023-01-11T20:48:01.1780536Z dist init r=1, world=4
2023-01-11T20:48:01.1780697Z dist init r=2, world=4
2023-01-11T20:48:01.1780900Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1781295Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1781782Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2066
2023-01-11T20:48:01.1782170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2067
2023-01-11T20:48:01.1782496Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2068
2023-01-11T20:48:01.1782816Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2069
2023-01-11T20:48:01.1783248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1783572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1783992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1784374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1784691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1785154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1785477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1785889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1786231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1786562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1787029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1787368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1787791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1788133Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1788453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1788943Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1789435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1789758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1790171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1790518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1790851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1791344Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1791851Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1792363Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1792748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1793093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1793426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1793766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1794025Z dist init r=2, world=4
2023-01-11T20:48:01.1794191Z dist init r=1, world=4
2023-01-11T20:48:01.1794368Z dist init r=0, world=4
2023-01-11T20:48:01.1794541Z dist init r=3, world=4
2023-01-11T20:48:01.1794734Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1795125Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1795621Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2122
2023-01-11T20:48:01.1796011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2123
2023-01-11T20:48:01.1796321Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2124
2023-01-11T20:48:01.1796665Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2125
2023-01-11T20:48:01.1797118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1797434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1797857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1798194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1798524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1798973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1799297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1799750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1800091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1800407Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1800869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1801190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1801596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1801930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1802261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1802761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1803270Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1803759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1804087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1804512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1804840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1805168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1805655Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1806158Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1806544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1806891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1807229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1807556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1807806Z dist init r=2, world=4
2023-01-11T20:48:01.1807986Z dist init r=3, world=4
2023-01-11T20:48:01.1808152Z dist init r=0, world=4
2023-01-11T20:48:01.1808325Z dist init r=1, world=4
2023-01-11T20:48:01.1808528Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1808908Z   test_optim_state_dict_nested_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1809437Z Tests :meth:`full_optim_state_dict` and meth:`sharded_optim_state_dict` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2178
2023-01-11T20:48:01.1809829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2179
2023-01-11T20:48:01.1810204Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2180
2023-01-11T20:48:01.1810514Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2181
2023-01-11T20:48:01.1811127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1811467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1811897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1812226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1812621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1813080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1813402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1813814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1814155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1814487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1815061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1815385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1815807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1816206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1816528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1817022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1817539Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1818034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1818347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1818768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1819112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1819429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1819917Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1820430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1820810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1821143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1821477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1821814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1822122Z dist init r=3, world=4
2023-01-11T20:48:01.1822305Z dist init r=0, world=4
2023-01-11T20:48:01.1822488Z dist init r=1, world=4
2023-01-11T20:48:01.1822666Z dist init r=2, world=4
2023-01-11T20:48:01.1822859Z skip: Need at least 2 CUDA devices (2.019s)
2023-01-11T20:48:01.1823215Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1823673Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2234
2023-01-11T20:48:01.1824042Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2235
2023-01-11T20:48:01.1824361Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2236
2023-01-11T20:48:01.1824682Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2237
2023-01-11T20:48:01.1825172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1825488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1825914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1826258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1826577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1827038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1827358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1827778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1828105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1828439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1828904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1829228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1829635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1829973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1830302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1830754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1831075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1831499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1831838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1832153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1832640Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1833153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1833666Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1834160Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1834541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1834920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1835248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1835589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1835842Z dist init r=0, world=4
2023-01-11T20:48:01.1836022Z dist init r=1, world=4
2023-01-11T20:48:01.1836182Z dist init r=3, world=4
2023-01-11T20:48:01.1836355Z dist init r=2, world=4
2023-01-11T20:48:01.1836560Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1836899Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_FULL_STATE_DICT_use_multiple_param_groups_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1837345Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2290
2023-01-11T20:48:01.1837774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2291
2023-01-11T20:48:01.1838084Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2292
2023-01-11T20:48:01.1838398Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2293
2023-01-11T20:48:01.1838845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1839174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1839590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1839932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1840270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1840735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1841049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1841478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1841821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1842140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1842606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1842938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1843367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1843694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1844031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1844525Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1845019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1845330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1845754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1846096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1846412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1846910Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1847463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1847983Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1848355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1848703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1849050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1849399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1849644Z dist init r=2, world=4
2023-01-11T20:48:01.1849821Z dist init r=3, world=4
2023-01-11T20:48:01.1849995Z dist init r=1, world=4
2023-01-11T20:48:01.1850156Z dist init r=0, world=4
2023-01-11T20:48:01.1850395Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1850750Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1851329Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2346
2023-01-11T20:48:01.1851708Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2347
2023-01-11T20:48:01.1852033Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2348
2023-01-11T20:48:01.1852353Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2349
2023-01-11T20:48:01.1852797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1853126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1853557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1853883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1854214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1854679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1855002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1855409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1855750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1856171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1856637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1856953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1857374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1857714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1858031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1858494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1858816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1859235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1859558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1859960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1860452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1860969Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1861470Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1861982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1862368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1862702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1863083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1863418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1863676Z dist init r=2, world=4
2023-01-11T20:48:01.1863844Z dist init r=3, world=4
2023-01-11T20:48:01.1864022Z dist init r=1, world=4
2023-01-11T20:48:01.1864196Z dist init r=0, world=4
2023-01-11T20:48:01.1864387Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1864735Z   test_rekey_optim_state_dict_to_ids_state_dict_type_StateDictType_SHARDED_STATE_DICT_use_multiple_param_groups_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1865191Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2402
2023-01-11T20:48:01.1865574Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2403
2023-01-11T20:48:01.1865885Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2404
2023-01-11T20:48:01.1866204Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2405
2023-01-11T20:48:01.1866655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1866967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1867392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1867729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1868061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1868510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1868831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1869258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1869584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1869913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1870374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1870695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1871102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1871443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1871769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1872228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1872570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1872990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1873326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1873640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1874131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1874644Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1875154Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1875686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1876069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1876412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1876747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1877074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1877326Z dist init r=0, world=4
2023-01-11T20:48:01.1877503Z dist init r=3, world=4
2023-01-11T20:48:01.1877663Z dist init r=2, world=4
2023-01-11T20:48:01.1877838Z dist init r=1, world=4
2023-01-11T20:48:01.1878041Z skip: Need at least 2 CUDA devices (1.920s)
2023-01-11T20:48:01.1878295Z   test_rekey_optim_state_dict_to_names (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1878675Z Tests :meth:`rekey_optim_state_dict` with the new keys being ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2458
2023-01-11T20:48:01.1879053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2459
2023-01-11T20:48:01.1879379Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2460
2023-01-11T20:48:01.1879685Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2461
2023-01-11T20:48:01.1880126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1880453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1880865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1881205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1881536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1881999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1882306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1882729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1883068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1883386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1883844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1884165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1884587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1884949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1885278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1885739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1886057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1886461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1886798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1887126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1887637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1888156Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1888672Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1889183Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1889550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1889894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1890229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1890567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1890921Z dist init r=2, world=4
2023-01-11T20:48:01.1891132Z dist init r=3, world=4
2023-01-11T20:48:01.1891310Z dist init r=0, world=4
2023-01-11T20:48:01.1891475Z dist init r=1, world=4
2023-01-11T20:48:01.1891681Z skip: Need at least 2 CUDA devices (1.920s)
2023-01-11T20:48:01.1892009Z   test_save_load_without_0th_param_state_state_dict_type_StateDictType_FULL_STATE_DICT (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1892433Z Tests saving and loading an optim state dict for Adam optimizer (i.e. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2514
2023-01-11T20:48:01.1892831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2515
2023-01-11T20:48:01.1893159Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2516
2023-01-11T20:48:01.1893488Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2517
2023-01-11T20:48:01.1893932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1894264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1894692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1895036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1895355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1895821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1896213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1896627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1896972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1897307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1897829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1898144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1898572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1898912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1899229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1899691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1900017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1900480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1900810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1901143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1901637Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1902155Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1902658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1903171Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1903554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1903905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1904052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1904210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1904285Z dist init r=3, world=4
2023-01-11T20:48:01.1904358Z dist init r=0, world=4
2023-01-11T20:48:01.1904430Z dist init r=2, world=4
2023-01-11T20:48:01.1904501Z dist init r=1, world=4
2023-01-11T20:48:01.1904603Z skip: Need at least 2 CUDA devices (2.020s)
2023-01-11T20:48:01.1904789Z   test_save_load_without_0th_param_state_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1905014Z Tests saving and loading an optim state dict for Adam optimizer (i.e. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2570
2023-01-11T20:48:01.1905169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2571
2023-01-11T20:48:01.1905318Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2572
2023-01-11T20:48:01.1905461Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2573
2023-01-11T20:48:01.1905737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1905861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1906141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1906274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1906436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1906705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1906864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1907144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1907279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1907454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1907716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1907838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1908099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1908230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1908475Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1908775Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1909044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1909165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1909443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1909576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1909751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1910032Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1910328Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1910629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1910793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1910955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1911111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1911267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1911343Z dist init r=3, world=4
2023-01-11T20:48:01.1911402Z dist init r=0, world=4
2023-01-11T20:48:01.1911475Z dist init r=1, world=4
2023-01-11T20:48:01.1911546Z dist init r=2, world=4
2023-01-11T20:48:01.1911648Z skip: Need at least 2 CUDA devices (1.920s)
2023-01-11T20:48:01.1911815Z   test_scatter_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1912143Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2626
2023-01-11T20:48:01.1912296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2627
2023-01-11T20:48:01.1912443Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2628
2023-01-11T20:48:01.1912573Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2629
2023-01-11T20:48:01.1912847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1912967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1913245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1913413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1913585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1913853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1913974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1914253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1914374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1914548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1914815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1914979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1915257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1915389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1915562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1915827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1915934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1916209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1916338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1916509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1916810Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1917107Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1917401Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1917694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1917856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1918005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1918163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1918321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1918398Z dist init r=0, world=4
2023-01-11T20:48:01.1918469Z dist init r=2, world=4
2023-01-11T20:48:01.1918539Z dist init r=3, world=4
2023-01-11T20:48:01.1918609Z dist init r=1, world=4
2023-01-11T20:48:01.1918711Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1918927Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1919257Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2682
2023-01-11T20:48:01.1919410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2683
2023-01-11T20:48:01.1919558Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2684
2023-01-11T20:48:01.1919702Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2685
2023-01-11T20:48:01.1920004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1920125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1920405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1920527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1920703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1920967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1921088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1921365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1921531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1921708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1921975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1922098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1922357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1922487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1922661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1922924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1923046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1923319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1923448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1923619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1923917Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1924200Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1924492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1924784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1924953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1925114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1925269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1925424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1925499Z dist init r=0, world=4
2023-01-11T20:48:01.1925559Z dist init r=2, world=4
2023-01-11T20:48:01.1925629Z dist init r=3, world=4
2023-01-11T20:48:01.1925700Z dist init r=1, world=4
2023-01-11T20:48:01.1925802Z skip: Need at least 2 CUDA devices (2.019s)
2023-01-11T20:48:01.1926026Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1926356Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2738
2023-01-11T20:48:01.1926541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2739
2023-01-11T20:48:01.1926688Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2740
2023-01-11T20:48:01.1926818Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2741
2023-01-11T20:48:01.1927092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1927215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1927496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1927630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1927803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1928101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1928223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1928504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1928626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1928802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1929071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1929192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1929470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1929605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1929779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1930076Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1930340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1930448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1930723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1931034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1931216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1931516Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1931813Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1932107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1932271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1932432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1932575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1932734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1932810Z dist init r=2, world=4
2023-01-11T20:48:01.1932883Z dist init r=0, world=4
2023-01-11T20:48:01.1933011Z dist init r=3, world=4
2023-01-11T20:48:01.1933082Z dist init r=1, world=4
2023-01-11T20:48:01.1933184Z skip: Need at least 2 CUDA devices (2.020s)
2023-01-11T20:48:01.1933397Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1933732Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2794
2023-01-11T20:48:01.1933884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2795
2023-01-11T20:48:01.1934030Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2796
2023-01-11T20:48:01.1934174Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2797
2023-01-11T20:48:01.1934448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1934607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1934893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1935029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1935191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1935459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1935583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1935858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1936058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1936234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1936507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1936629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1936891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1937023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1937194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1937460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1937582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1937858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1937996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1938166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1938463Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1938745Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1939040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1939331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1939494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1939659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1939855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1940007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1940082Z dist init r=1, world=4
2023-01-11T20:48:01.1940141Z dist init r=3, world=4
2023-01-11T20:48:01.1940212Z dist init r=0, world=4
2023-01-11T20:48:01.1940282Z dist init r=2, world=4
2023-01-11T20:48:01.1940385Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1940611Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1940942Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2850
2023-01-11T20:48:01.1941095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2851
2023-01-11T20:48:01.1941272Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2852
2023-01-11T20:48:01.1941404Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2853
2023-01-11T20:48:01.1941676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1941800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1942082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1942219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1942400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1942698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1942975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1943096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1943364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1943498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1943672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1943938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1944058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1944331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1944463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1944637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1944909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1945019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1945294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1945425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1945597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1945892Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1946190Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1946514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1946679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1946840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1946987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1947142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1947217Z dist init r=1, world=4
2023-01-11T20:48:01.1947290Z dist init r=3, world=4
2023-01-11T20:48:01.1947363Z dist init r=2, world=4
2023-01-11T20:48:01.1947433Z dist init r=0, world=4
2023-01-11T20:48:01.1947535Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1947776Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1948104Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2906
2023-01-11T20:48:01.1948257Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2907
2023-01-11T20:48:01.1948409Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2908
2023-01-11T20:48:01.1948556Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2909
2023-01-11T20:48:01.1948834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1948957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1949238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1949377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1949542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1949812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1949933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1950212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1950346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1950522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1950791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1950918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1951180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1951316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1951491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1951761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1951884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1952158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1952294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1952469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1952799Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1953080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1953374Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1953668Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1953834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1954001Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1954157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1954345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1954423Z dist init r=3, world=4
2023-01-11T20:48:01.1954494Z dist init r=0, world=4
2023-01-11T20:48:01.1954553Z dist init r=1, world=4
2023-01-11T20:48:01.1954625Z dist init r=2, world=4
2023-01-11T20:48:01.1954730Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1954955Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1955283Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2962
2023-01-11T20:48:01.1955438Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2963
2023-01-11T20:48:01.1955589Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2964
2023-01-11T20:48:01.1955736Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2965
2023-01-11T20:48:01.1955997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1956121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1956402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1956537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1956711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1956984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1957107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1957390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1957512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1957689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1957959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1958083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1958360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1958491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1958663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1958960Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1959261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1959371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1959646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1959780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1959954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1960250Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1960543Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1960880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1961048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1961214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1961359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1961519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1961592Z dist init r=0, world=4
2023-01-11T20:48:01.1961663Z dist init r=1, world=4
2023-01-11T20:48:01.1961736Z dist init r=2, world=4
2023-01-11T20:48:01.1961808Z dist init r=3, world=4
2023-01-11T20:48:01.1961911Z skip: Need at least 2 CUDA devices (2.019s)
2023-01-11T20:48:01.1962119Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1962449Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3018
2023-01-11T20:48:01.1962603Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3019
2023-01-11T20:48:01.1962749Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3020
2023-01-11T20:48:01.1962893Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3021
2023-01-11T20:48:01.1963166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1963289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1963570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1963704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1963867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1964135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1964255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1964528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1964661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1964833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1965098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1965216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1965485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1965646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1965821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1966090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1966208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1966487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1966617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1966789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1967085Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1967400Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1967696Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1967991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1968153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1968312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1968467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1968624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1968696Z dist init r=0, world=4
2023-01-11T20:48:01.1968772Z dist init r=3, world=4
2023-01-11T20:48:01.1968830Z dist init r=1, world=4
2023-01-11T20:48:01.1968903Z dist init r=2, world=4
2023-01-11T20:48:01.1969006Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1969229Z   test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1969555Z Tests :meth:`scatter_full_optim_state_dict` for a non-FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3074
2023-01-11T20:48:01.1969707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3075
2023-01-11T20:48:01.1969855Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3076
2023-01-11T20:48:01.1969998Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3077
2023-01-11T20:48:01.1970256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1970381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1970661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1970795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1971101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1971369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1971492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1971767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1971886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1972064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1972385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1972506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1972782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1972913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1973086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1973379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1973648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1973791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1974075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1974208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1974385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1974684Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1974980Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1975270Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1975432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1975595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1975739Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1975893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1976028Z dist init r=2, world=4
2023-01-11T20:48:01.1976108Z dist init r=1, world=4
2023-01-11T20:48:01.1976181Z dist init r=0, world=4
2023-01-11T20:48:01.1976253Z dist init r=3, world=4
2023-01-11T20:48:01.1976356Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1976496Z   test_scatter_full_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1976820Z Tests :meth:`scatter_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3130
2023-01-11T20:48:01.1976974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3131
2023-01-11T20:48:01.1977126Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3132
2023-01-11T20:48:01.1977277Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3133
2023-01-11T20:48:01.1977553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1977675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1977954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1978090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1978253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1978522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1978646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1978963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1979094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1979265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1979533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1979652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1979914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1980045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1980215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1980508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1980630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1980902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1981033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1981208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1981503Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1981783Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1982080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1982375Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1982537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1982699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1982858Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1983014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1983088Z dist init r=1, world=4
2023-01-11T20:48:01.1983160Z dist init r=2, world=4
2023-01-11T20:48:01.1983219Z dist init r=0, world=4
2023-01-11T20:48:01.1983290Z dist init r=3, world=4
2023-01-11T20:48:01.1983392Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1983555Z   test_shard_full_optim_state_dict_nested_halve_world_size (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1983894Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3186
2023-01-11T20:48:01.1984046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3187
2023-01-11T20:48:01.1984192Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3188
2023-01-11T20:48:01.1984323Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3189
2023-01-11T20:48:01.1984595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1984719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1984999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1985134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1985343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1985612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1985733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1986009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1986128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1986301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1986566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1986685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1986989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1987120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1987289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1987553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1987659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1987931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1988060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1988228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1988527Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1988823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1989114Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1989406Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1989566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1989729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1989873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1990030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1990108Z dist init r=3, world=4
2023-01-11T20:48:01.1990181Z dist init r=1, world=4
2023-01-11T20:48:01.1990252Z dist init r=2, world=4
2023-01-11T20:48:01.1990323Z dist init r=0, world=4
2023-01-11T20:48:01.1990425Z skip: Need at least 2 CUDA devices (1.920s)
2023-01-11T20:48:01.1990636Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1990964Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3242
2023-01-11T20:48:01.1991115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3243
2023-01-11T20:48:01.1991262Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3244
2023-01-11T20:48:01.1991404Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3245
2023-01-11T20:48:01.1991681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1991832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1992113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1992247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1992410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1992678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1992797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1993072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1993233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1993409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.1993680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1993801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1994066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1994197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1994371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.1994635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1994756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1995035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1995166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1995338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.1995633Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1995914Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1996206Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1996497Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.1996661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.1996827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.1996984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.1997139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.1997212Z dist init r=1, world=4
2023-01-11T20:48:01.1997272Z dist init r=3, world=4
2023-01-11T20:48:01.1997342Z dist init r=2, world=4
2023-01-11T20:48:01.1997413Z dist init r=0, world=4
2023-01-11T20:48:01.1997513Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.1997737Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.1998073Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3298
2023-01-11T20:48:01.1998252Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3299
2023-01-11T20:48:01.1998398Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3300
2023-01-11T20:48:01.1998528Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3301
2023-01-11T20:48:01.1998798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1998922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.1999200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.1999335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.1999510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.1999829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.1999950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2000231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2000351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2024186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.2024577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2024706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2024997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2025142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2025322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.2025593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2025714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2025993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2026113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2026293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.2026593Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2026898Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2027198Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2027491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2027659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.2027821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.2027982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.2028124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.2028199Z dist init r=2, world=4
2023-01-11T20:48:01.2028271Z dist init r=1, world=4
2023-01-11T20:48:01.2028344Z dist init r=0, world=4
2023-01-11T20:48:01.2028554Z dist init r=3, world=4
2023-01-11T20:48:01.2028660Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.2028885Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.2029225Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3354
2023-01-11T20:48:01.2029367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3355
2023-01-11T20:48:01.2029520Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3356
2023-01-11T20:48:01.2029663Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3357
2023-01-11T20:48:01.2029936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2030060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2030401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2030538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2030706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.2030969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2031089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2031367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2031500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2031676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.2031947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2032071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2032347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2032477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2032637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.2032938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2033204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2033324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2033604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2033738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2033907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.2034203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2034492Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2034773Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2034935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.2035088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.2035310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.2035467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.2035535Z dist init r=0, world=4
2023-01-11T20:48:01.2035600Z dist init r=1, world=4
2023-01-11T20:48:01.2035663Z dist init r=3, world=4
2023-01-11T20:48:01.2035721Z dist init r=2, world=4
2023-01-11T20:48:01.2035818Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.2036040Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.2036366Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3410
2023-01-11T20:48:01.2036519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3411
2023-01-11T20:48:01.2036694Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3412
2023-01-11T20:48:01.2036838Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3413
2023-01-11T20:48:01.2037106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2037215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2037493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2037628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2037797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.2038064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2038181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2038461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2038587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2038755Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.2039011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2039131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2039399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2039524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2039693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.2039957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2040069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2040342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2040462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2040626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.2040918Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2041204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2041487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2041813Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2041970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.2042131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.2042283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.2042439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.2042501Z dist init r=1, world=4
2023-01-11T20:48:01.2042567Z dist init r=0, world=4
2023-01-11T20:48:01.2042630Z dist init r=2, world=4
2023-01-11T20:48:01.2042695Z dist init r=3, world=4
2023-01-11T20:48:01.2042796Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.2043037Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.2043363Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3466
2023-01-11T20:48:01.2043503Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3467
2023-01-11T20:48:01.2043650Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3468
2023-01-11T20:48:01.2043791Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3469
2023-01-11T20:48:01.2044056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2044169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2044438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2044573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2044739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.2045001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2045110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2045386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2045513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2045687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.2045951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2046067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2046344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2046470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2046629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.2046923Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2047181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2047295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2047571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2047697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2047898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.2048187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2048474Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2048755Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2048915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.2049069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.2049226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.2049406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.2049483Z dist init r=0, world=4
2023-01-11T20:48:01.2049549Z dist init r=3, world=4
2023-01-11T20:48:01.2049618Z dist init r=2, world=4
2023-01-11T20:48:01.2049676Z dist init r=1, world=4
2023-01-11T20:48:01.2049773Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.2049990Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.2050316Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3522
2023-01-11T20:48:01.2050467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3523
2023-01-11T20:48:01.2050611Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3524
2023-01-11T20:48:01.2050757Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3525
2023-01-11T20:48:01.2051161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2051270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2051546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2051675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2051846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.2052110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2052228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2052498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2052634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2052802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.2053055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2053168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2053442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2053566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2053730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.2054021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2054310Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2054622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2054736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2055001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2055126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2055295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.2055581Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2055869Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2056131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.2056290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.2056444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.2056600Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.2056663Z dist init r=1, world=4
2023-01-11T20:48:01.2056730Z dist init r=3, world=4
2023-01-11T20:48:01.2056802Z dist init r=0, world=4
2023-01-11T20:48:01.2056873Z dist init r=2, world=4
2023-01-11T20:48:01.2056976Z skip: Need at least 2 CUDA devices (1.920s)
2023-01-11T20:48:01.2057194Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.2057535Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3578
2023-01-11T20:48:01.2057677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3579
2023-01-11T20:48:01.2057828Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3580
2023-01-11T20:48:01.2057977Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3581
2023-01-11T20:48:01.2058250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2058371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2058649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2058780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2058951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.2059223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2059331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2059609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2059739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2059914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.2060183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2060303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2060578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2060746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2060906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.2061202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2061468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2061591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2061871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2062004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2062174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.2062495Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2062790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2063070Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2063233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.2063393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.2063553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.2063711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.2063786Z dist init r=0, world=4
2023-01-11T20:48:01.2063860Z dist init r=1, world=4
2023-01-11T20:48:01.2063937Z dist init r=2, world=4
2023-01-11T20:48:01.2063997Z dist init r=3, world=4
2023-01-11T20:48:01.2064101Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.2064320Z   test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.2064650Z Tests :meth:`shard_full_optim_state_dict` for a non-FSDP-root model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3634
2023-01-11T20:48:01.2064802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3635
2023-01-11T20:48:01.2064954Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3636
2023-01-11T20:48:01.2065103Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3637
2023-01-11T20:48:01.2065374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2065500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2065766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2065899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2066073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.2066343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2066465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2066739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2066871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2067047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.2067332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2067454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2067727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2067858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2068031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.2068329Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2068597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2068715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2069021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2069141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2069311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.2069603Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2069895Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2070186Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2070349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.2070514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.2070675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.2070830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.2070892Z dist init r=0, world=4
2023-01-11T20:48:01.2070962Z dist init r=2, world=4
2023-01-11T20:48:01.2071034Z dist init r=3, world=4
2023-01-11T20:48:01.2071103Z dist init r=1, world=4
2023-01-11T20:48:01.2071207Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.2071357Z   test_shard_full_optim_state_dict_transformer (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.2071672Z Tests :meth:`shard_full_optim_state_dict` for an FSDP-root ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3690
2023-01-11T20:48:01.2071812Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3691
2023-01-11T20:48:01.2071962Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3692
2023-01-11T20:48:01.2072114Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3693
2023-01-11T20:48:01.2072387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2072508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2072786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2072919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2073092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.2073362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2073469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2073782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2073916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2074091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.2074359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2074481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2074758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2074891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2075049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.2075386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2075657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2075778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2076058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2076191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2076361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.2076653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2076945Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2077230Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2077394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.2077554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.2077714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.2077872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.2077946Z dist init r=0, world=4
2023-01-11T20:48:01.2078018Z dist init r=2, world=4
2023-01-11T20:48:01.2078088Z dist init r=3, world=4
2023-01-11T20:48:01.2078146Z dist init r=1, world=4
2023-01-11T20:48:01.2078250Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.2078485Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.2078712Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3746
2023-01-11T20:48:01.2078859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3747
2023-01-11T20:48:01.2079002Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3748
2023-01-11T20:48:01.2079146Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3749
2023-01-11T20:48:01.2079417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2079525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2079803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2079940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2080145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.2080415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2080537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2080816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2080947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2081125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.2081376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2081497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2081804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2081939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2082113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.2082383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2082505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2082782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2082914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2083076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.2083377Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2083675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2083965Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2084257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2084418Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.2084581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.2084740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.2084903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.2084967Z dist init r=0, world=4
2023-01-11T20:48:01.2085039Z dist init r=3, world=4
2023-01-11T20:48:01.2085111Z dist init r=1, world=4
2023-01-11T20:48:01.2085183Z dist init r=2, world=4
2023-01-11T20:48:01.2085287Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.2085522Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_FULL_STATE_DICT_add_to_fsdp_module_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.2085746Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3802
2023-01-11T20:48:01.2085886Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3803
2023-01-11T20:48:01.2086039Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3804
2023-01-11T20:48:01.2086188Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3805
2023-01-11T20:48:01.2086497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2086619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2086897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2087030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2087202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.2087472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2087580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2087856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2088019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2088193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.2088461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2088583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2088862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2088993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2089154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.2089422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2089545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2089823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2089956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2090130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.2090430Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2090725Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2091161Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2091441Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2091609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.2091773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.2091934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.2092094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.2092168Z dist init r=2, world=4
2023-01-11T20:48:01.2092242Z dist init r=0, world=4
2023-01-11T20:48:01.2092314Z dist init r=3, world=4
2023-01-11T20:48:01.2092373Z dist init r=1, world=4
2023-01-11T20:48:01.2092476Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.2092714Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_False (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.2092939Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3858
2023-01-11T20:48:01.2093138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3859
2023-01-11T20:48:01.2093287Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3860
2023-01-11T20:48:01.2093442Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3861
2023-01-11T20:48:01.2093718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2093828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2094111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2094244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2094420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.2094728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2094850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2095131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2095263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2095425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.2095690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2095810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2096147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2096287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2096460Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.2096758Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2097025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2097146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2097413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2097548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2097721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.2098016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2098313Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2098605Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2098768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.2098932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.2099090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.2099238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.2099313Z dist init r=0, world=4
2023-01-11T20:48:01.2099386Z dist init r=2, world=4
2023-01-11T20:48:01.2099494Z dist init r=1, world=4
2023-01-11T20:48:01.2099567Z dist init r=3, world=4
2023-01-11T20:48:01.2099671Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.2099906Z   test_shard_full_optim_state_dict_unmanaged_params_state_dict_type_StateDictType_SHARDED_STATE_DICT_add_to_fsdp_module_True (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.2100118Z Tests :meth:`shard_full_optim_state_dict` when there are unmanaged ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3914
2023-01-11T20:48:01.2100269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3915
2023-01-11T20:48:01.2100419Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3916
2023-01-11T20:48:01.2100569Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3917
2023-01-11T20:48:01.2100844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2100996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2101279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2101412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2101587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.2101846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2101964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2102242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2102374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2102551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.2102819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2102938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2103212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2103343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2103503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.2103768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2103887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2104161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2104296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2104467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.2104760Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2105053Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2105327Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2105620Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2105780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.2105943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.2106136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.2106294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.2106370Z dist init r=3, world=4
2023-01-11T20:48:01.2106441Z dist init r=0, world=4
2023-01-11T20:48:01.2106500Z dist init r=1, world=4
2023-01-11T20:48:01.2106569Z dist init r=2, world=4
2023-01-11T20:48:01.2106671Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.2106792Z   test_use_orig_params (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.2107054Z Tests :meth:`optim_state_dict` for an FSDP-root nested model. ... skip: The test currently fails on CI. (0.001s)
2023-01-11T20:48:01.2107180Z   test_use_orig_params_error (__main__.TestFSDPOptimState)
2023-01-11T20:48:01.2107441Z Tests that the optimizer state checkpointing APIs raise an error ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3970
2023-01-11T20:48:01.2107597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3971
2023-01-11T20:48:01.2107735Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3972
2023-01-11T20:48:01.2107883Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3973
2023-01-11T20:48:01.2108154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2108274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2108552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2108686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2108859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:48:01.2109132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2109251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2109518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2109649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2109820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:48:01.2110081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2110201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2110475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2110610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2110784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:48:01.2111067Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2111331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:01.2111450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:01.2111726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:01.2111858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:01.2112024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:48:01.2112317Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2112640Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2112932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:48:01.2113093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:48:01.2113244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:48:01.2113401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:48:01.2113559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:48:01.2113632Z dist init r=0, world=4
2023-01-11T20:48:01.2113704Z dist init r=2, world=4
2023-01-11T20:48:01.2113774Z dist init r=1, world=4
2023-01-11T20:48:01.2113872Z dist init r=3, world=4
2023-01-11T20:48:01.2113964Z skip: Need at least 2 CUDA devices (1.919s)
2023-01-11T20:48:01.2113972Z 
2023-01-11T20:48:01.2114168Z ----------------------------------------------------------------------
2023-01-11T20:48:01.2114244Z Ran 55 tests in 102.803s
2023-01-11T20:48:01.2114249Z 
2023-01-11T20:48:01.2114321Z OK (skipped=55)
2023-01-11T20:48:01.2114326Z 
2023-01-11T20:48:01.2114408Z Generating XML reports...
2023-01-11T20:48:01.2114750Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20230111204617.xml
2023-01-11T20:48:01.2114755Z 
2023-01-11T20:48:01.2115212Z ##[endgroup]
2023-01-11T20:48:01.2115578Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_optim_state (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_optim_state_v5kxh7_d)
2023-01-11T20:48:01.2115584Z 
2023-01-11T20:48:01.2115813Z Running distributed/algorithms/quantization/test_quantization ... [2023-01-11 20:48:01.150047]
2023-01-11T20:48:01.2115971Z MPI not available -- MPI backend tests will be skipped
2023-01-11T20:48:01.2116260Z Map different backends to different shards for distributed/algorithms/quantization/test_quantization: {'gloo': 1, 'nccl': 2}
2023-01-11T20:48:01.2116349Z Shard 2: test should be run in 1
2023-01-11T20:48:01.2116438Z Shard 2: gloo should be run in 1
2023-01-11T20:48:01.2116635Z Running distributed/test_distributed_spawn ... [2023-01-11 20:48:01.156855]
2023-01-11T20:48:01.2116825Z MPI not available -- MPI backend tests will be skipped
2023-01-11T20:48:01.2117094Z Map different backends to different shards for distributed/test_distributed_spawn: {'gloo': 1, 'nccl': 2, 'ucc': 1}
2023-01-11T20:48:01.2117178Z Shard 2: test should be run in 1
2023-01-11T20:48:01.2117249Z Shard 2: gloo should be run in 1
2023-01-11T20:48:01.2117451Z Running distributed/pipeline/sync/test_worker ... [2023-01-11 20:48:01.161180]
2023-01-11T20:48:01.2117748Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_worker.py', '-v'] ... [2023-01-11 20:48:01.161481]
2023-01-11T20:48:03.3030506Z 
2023-01-11T20:48:03.3031385Z Expand the folded group to see the log file of distributed/pipeline/sync/test_worker
2023-01-11T20:48:03.3032335Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_worker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_worker_nn_d51jb)
2023-01-11T20:48:03.3032722Z ============================= test session starts ==============================
2023-01-11T20:48:03.3033182Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T20:48:03.3033458Z cachedir: .pytest_cache
2023-01-11T20:48:03.3033878Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T20:48:03.3034319Z torch: 2.0.0a0+git8419ddd
2023-01-11T20:48:03.3034562Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T20:48:03.3035283Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T20:48:03.3035699Z collecting ... collected 6 items
2023-01-11T20:48:03.3036635Z Running 6 items in this shard: test/distributed/pipeline/sync/test_worker.py::test_compute_multithreading, test/distributed/pipeline/sync/test_worker.py::test_compute_success, test/distributed/pipeline/sync/test_worker.py::test_compute_exception, test/distributed/pipeline/sync/test_worker.py::test_grad_mode[True], test/distributed/pipeline/sync/test_worker.py::test_grad_mode[False], test/distributed/pipeline/sync/test_worker.py::test_worker_per_device
2023-01-11T20:48:03.3037171Z 
2023-01-11T20:48:03.3037333Z distributed/pipeline/sync/test_worker.py::test_compute_multithreading PASSED [ 16%]
2023-01-11T20:48:03.3037778Z distributed/pipeline/sync/test_worker.py::test_compute_success PASSED    [ 33%]
2023-01-11T20:48:03.3038091Z distributed/pipeline/sync/test_worker.py::test_compute_exception PASSED  [ 50%]
2023-01-11T20:48:03.3038595Z distributed/pipeline/sync/test_worker.py::test_grad_mode[True] PASSED    [ 66%]
2023-01-11T20:48:03.3039203Z distributed/pipeline/sync/test_worker.py::test_grad_mode[False] PASSED   [ 83%]
2023-01-11T20:48:03.3039527Z distributed/pipeline/sync/test_worker.py::test_worker_per_device PASSED  [100%]
2023-01-11T20:48:03.3039692Z 
2023-01-11T20:48:03.3039801Z ============================== 6 passed in 0.04s ===============================
2023-01-11T20:48:03.3039935Z 
2023-01-11T20:48:03.3040177Z ##[endgroup]
2023-01-11T20:48:03.3040661Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_worker (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_worker_nn_d51jb)
2023-01-11T20:48:03.3040922Z 
2023-01-11T20:48:03.3041136Z Running distributed/pipeline/sync/test_stream ... [2023-01-11 20:48:03.303355]
2023-01-11T20:48:03.3041589Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_stream.py', '-v'] ... [2023-01-11 20:48:03.303710]
2023-01-11T20:48:04.9756655Z 
2023-01-11T20:48:04.9757174Z Expand the folded group to see the log file of distributed/pipeline/sync/test_stream
2023-01-11T20:48:04.9758278Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_stream (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_stream_xutiqnwt)
2023-01-11T20:48:04.9758668Z ============================= test session starts ==============================
2023-01-11T20:48:04.9759141Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T20:48:04.9759382Z cachedir: .pytest_cache
2023-01-11T20:48:04.9759813Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T20:48:04.9760211Z torch: 2.0.0a0+git8419ddd
2023-01-11T20:48:04.9760535Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T20:48:04.9760982Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T20:48:04.9761283Z collecting ... collected 19 items
2023-01-11T20:48:04.9763230Z Running 19 items in this shard: test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu, test/distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu, test/distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda, test/distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view
2023-01-11T20:48:04.9765364Z 
2023-01-11T20:48:04.9765538Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cpu PASSED [  5%]
2023-01-11T20:48:04.9765966Z distributed/pipeline/sync/test_stream.py::TestNewStream::test_new_stream_cuda SKIPPED [ 10%]
2023-01-11T20:48:04.9766323Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cpu PASSED [ 15%]
2023-01-11T20:48:04.9766700Z distributed/pipeline/sync/test_stream.py::TestCurrentStream::test_current_stream_cuda SKIPPED [ 21%]
2023-01-11T20:48:04.9767070Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cpu PASSED [ 26%]
2023-01-11T20:48:04.9767438Z distributed/pipeline/sync/test_stream.py::TestDefaultStream::test_default_stream_cuda SKIPPED [ 31%]
2023-01-11T20:48:04.9767785Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cpu PASSED [ 36%]
2023-01-11T20:48:04.9768137Z distributed/pipeline/sync/test_stream.py::TestUseDevice::test_use_device_cuda SKIPPED [ 42%]
2023-01-11T20:48:04.9768487Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cpu PASSED [ 47%]
2023-01-11T20:48:04.9768822Z distributed/pipeline/sync/test_stream.py::TestUseStream::test_use_stream_cuda SKIPPED [ 52%]
2023-01-11T20:48:04.9769164Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cpu PASSED [ 57%]
2023-01-11T20:48:04.9769509Z distributed/pipeline/sync/test_stream.py::TestGetDevice::test_get_device_cuda SKIPPED [ 63%]
2023-01-11T20:48:04.9769862Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cpu PASSED [ 68%]
2023-01-11T20:48:04.9770212Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cpu_cuda SKIPPED [ 73%]
2023-01-11T20:48:04.9770573Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cpu SKIPPED [ 78%]
2023-01-11T20:48:04.9771105Z distributed/pipeline/sync/test_stream.py::TestWaitStream::test_wait_stream_cuda_cuda SKIPPED [ 84%]
2023-01-11T20:48:04.9771474Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cpu PASSED [ 89%]
2023-01-11T20:48:04.9771827Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_cuda SKIPPED [ 94%]
2023-01-11T20:48:04.9772214Z distributed/pipeline/sync/test_stream.py::TestRecordStream::test_record_stream_shifted_view SKIPPED [100%]
2023-01-11T20:48:04.9772424Z 
2023-01-11T20:48:04.9772544Z ======================== 8 passed, 11 skipped in 0.05s =========================
2023-01-11T20:48:04.9772688Z 
2023-01-11T20:48:04.9772944Z ##[endgroup]
2023-01-11T20:48:04.9773447Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_stream (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_stream_xutiqnwt)
2023-01-11T20:48:04.9773727Z 
2023-01-11T20:48:04.9773944Z Running distributed/pipeline/sync/test_pipe ... [2023-01-11 20:48:04.975934]
2023-01-11T20:48:04.9774403Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipe.py', '-v'] ... [2023-01-11 20:48:04.976258]
2023-01-11T20:48:08.2318679Z 
2023-01-11T20:48:08.2319247Z Expand the folded group to see the log file of distributed/pipeline/sync/test_pipe
2023-01-11T20:48:08.2323010Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_pipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipe_7zg672xr)
2023-01-11T20:48:08.2323679Z ============================= test session starts ==============================
2023-01-11T20:48:08.2324397Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T20:48:08.2324705Z cachedir: .pytest_cache
2023-01-11T20:48:08.2325138Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T20:48:08.2325449Z torch: 2.0.0a0+git8419ddd
2023-01-11T20:48:08.2325690Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T20:48:08.2326112Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T20:48:08.2326410Z collecting ... collected 56 items
2023-01-11T20:48:08.2331298Z Running 56 items in this shard: test/distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc, test/distributed/pipeline/sync/test_pipe.py::test_parameters, test/distributed/pipeline/sync/test_pipe.py::test_public_attrs, test/distributed/pipeline/sync/test_pipe.py::test_sequential_like, test/distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_small, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input, test/distributed/pipeline/sync/test_pipe.py::test_no_grad, test/distributed/pipeline/sync/test_pipe.py::test_exception, test/distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap, test/distributed/pipeline/sync/test_pipe.py::test_nested_input, test/distributed/pipeline/sync/test_pipe.py::test_input_pair, test/distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input, test/distributed/pipeline/sync/test_pipe.py::test_input_singleton, test/distributed/pipeline/sync/test_pipe.py::test_input_varargs, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence, test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[never], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[always], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always], test/distributed/pipeline/sync/test_pipe.py::test_devices, test/distributed/pipeline/sync/test_pipe.py::test_partitions, test/distributed/pipeline/sync/test_pipe.py::test_merged_partitions, test/distributed/pipeline/sync/test_pipe.py::test_deny_moving, test/distributed/pipeline/sync/test_pipe.py::test_empty_module, test/distributed/pipeline/sync/test_pipe.py::test_named_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_forward_lockstep, test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last], test/distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device, test/distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper
2023-01-11T20:48:08.2335621Z 
2023-01-11T20:48:08.2335809Z distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc PASSED     [  1%]
2023-01-11T20:48:08.2336222Z distributed/pipeline/sync/test_pipe.py::test_parameters PASSED           [  3%]
2023-01-11T20:48:08.2336530Z distributed/pipeline/sync/test_pipe.py::test_public_attrs PASSED         [  5%]
2023-01-11T20:48:08.2336841Z distributed/pipeline/sync/test_pipe.py::test_sequential_like PASSED      [  7%]
2023-01-11T20:48:08.2337228Z distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1 PASSED   [  8%]
2023-01-11T20:48:08.2337549Z distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible PASSED [ 10%]
2023-01-11T20:48:08.2337865Z distributed/pipeline/sync/test_pipe.py::test_batch_size_small PASSED     [ 12%]
2023-01-11T20:48:08.2338158Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode PASSED      [ 14%]
2023-01-11T20:48:08.2338479Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid PASSED [ 16%]
2023-01-11T20:48:08.2338818Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1 PASSED [ 17%]
2023-01-11T20:48:08.2339150Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval PASSED      [ 19%]
2023-01-11T20:48:08.2339463Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input PASSED [ 21%]
2023-01-11T20:48:08.2339780Z distributed/pipeline/sync/test_pipe.py::test_no_grad PASSED              [ 23%]
2023-01-11T20:48:08.2340079Z distributed/pipeline/sync/test_pipe.py::test_exception PASSED            [ 25%]
2023-01-11T20:48:08.2340386Z distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap PASSED [ 26%]
2023-01-11T20:48:08.2340707Z distributed/pipeline/sync/test_pipe.py::test_nested_input PASSED         [ 28%]
2023-01-11T20:48:08.2341015Z distributed/pipeline/sync/test_pipe.py::test_input_pair PASSED           [ 30%]
2023-01-11T20:48:08.2341322Z distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input PASSED [ 32%]
2023-01-11T20:48:08.2341623Z distributed/pipeline/sync/test_pipe.py::test_input_singleton PASSED      [ 33%]
2023-01-11T20:48:08.2341927Z distributed/pipeline/sync/test_pipe.py::test_input_varargs PASSED        [ 35%]
2023-01-11T20:48:08.2342233Z distributed/pipeline/sync/test_pipe.py::test_non_tensor PASSED           [ 37%]
2023-01-11T20:48:08.2342522Z distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence PASSED  [ 39%]
2023-01-11T20:48:08.2342842Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never] PASSED [ 41%]
2023-01-11T20:48:08.2343168Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always] PASSED [ 42%]
2023-01-11T20:48:08.2343499Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last] PASSED [ 44%]
2023-01-11T20:48:08.2343814Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never] PASSED [ 46%]
2023-01-11T20:48:08.2344136Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always] PASSED [ 48%]
2023-01-11T20:48:08.2344466Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last] PASSED [ 50%]
2023-01-11T20:48:08.2344831Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never] PASSED [ 51%]
2023-01-11T20:48:08.2345159Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always] PASSED [ 53%]
2023-01-11T20:48:08.2345491Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last] PASSED [ 55%]
2023-01-11T20:48:08.2345816Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[never] PASSED      [ 57%]
2023-01-11T20:48:08.2346115Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[always] PASSED     [ 58%]
2023-01-11T20:48:08.2346430Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last] PASSED [ 60%]
2023-01-11T20:48:08.2346754Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never] PASSED [ 62%]
2023-01-11T20:48:08.2347085Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always] PASSED [ 64%]
2023-01-11T20:48:08.2347411Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last] PASSED [ 66%]
2023-01-11T20:48:08.2347809Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never] PASSED [ 67%]
2023-01-11T20:48:08.2348379Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always] PASSED [ 69%]
2023-01-11T20:48:08.2348887Z distributed/pipeline/sync/test_pipe.py::test_devices PASSED              [ 71%]
2023-01-11T20:48:08.2349372Z distributed/pipeline/sync/test_pipe.py::test_partitions PASSED           [ 73%]
2023-01-11T20:48:08.2349861Z distributed/pipeline/sync/test_pipe.py::test_merged_partitions SKIPPED   [ 75%]
2023-01-11T20:48:08.2350316Z distributed/pipeline/sync/test_pipe.py::test_deny_moving PASSED          [ 76%]
2023-01-11T20:48:08.2350722Z distributed/pipeline/sync/test_pipe.py::test_empty_module PASSED         [ 78%]
2023-01-11T20:48:08.2351151Z distributed/pipeline/sync/test_pipe.py::test_named_children PASSED       [ 80%]
2023-01-11T20:48:08.2351657Z distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential PASSED [ 82%]
2023-01-11T20:48:08.2352219Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children PASSED [ 83%]
2023-01-11T20:48:08.2352780Z distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device SKIPPED [ 85%]
2023-01-11T20:48:08.2353315Z distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules SKIPPED [ 87%]
2023-01-11T20:48:08.2353881Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device PASSED [ 89%]
2023-01-11T20:48:08.2354414Z distributed/pipeline/sync/test_pipe.py::test_forward_lockstep PASSED     [ 91%]
2023-01-11T20:48:08.2354946Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never] SKIPPED [ 92%]
2023-01-11T20:48:08.2355492Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always] SKIPPED [ 94%]
2023-01-11T20:48:08.2356036Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last] SKIPPED [ 96%]
2023-01-11T20:48:08.2356548Z distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device SKIPPED [ 98%]
2023-01-11T20:48:08.2357026Z distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper SKIPPED [100%]
2023-01-11T20:48:08.2357293Z 
2023-01-11T20:48:08.2357458Z ======================== 48 passed, 8 skipped in 1.62s =========================
2023-01-11T20:48:08.2357659Z 
2023-01-11T20:48:08.2358036Z ##[endgroup]
2023-01-11T20:48:08.2358797Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_pipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipe_7zg672xr)
2023-01-11T20:48:08.2359199Z 
2023-01-11T20:48:08.2359543Z Running distributed/pipeline/sync/test_microbatch ... [2023-01-11 20:48:08.232246]
2023-01-11T20:48:08.2360282Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_microbatch.py', '-v'] ... [2023-01-11 20:48:08.232610]
2023-01-11T20:48:09.9241233Z 
2023-01-11T20:48:09.9241799Z Expand the folded group to see the log file of distributed/pipeline/sync/test_microbatch
2023-01-11T20:48:09.9242983Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_microbatch (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_microbatch_rh772dbe)
2023-01-11T20:48:09.9243403Z ============================= test session starts ==============================
2023-01-11T20:48:09.9243916Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T20:48:09.9244173Z cachedir: .pytest_cache
2023-01-11T20:48:09.9244792Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T20:48:09.9245377Z torch: 2.0.0a0+git8419ddd
2023-01-11T20:48:09.9245758Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T20:48:09.9246221Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T20:48:09.9246658Z collecting ... collected 10 items
2023-01-11T20:48:09.9247903Z Running 10 items in this shard: test/distributed/pipeline/sync/test_microbatch.py::test_batch_atomic, test/distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic, test/distributed/pipeline/sync/test_microbatch.py::test_batch_call, test/distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index, test/distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice, test/distributed/pipeline/sync/test_microbatch.py::test_check, test/distributed/pipeline/sync/test_microbatch.py::test_gather_tensors, test/distributed/pipeline/sync/test_microbatch.py::test_gather_tuples, test/distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor, test/distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors
2023-01-11T20:48:09.9248750Z 
2023-01-11T20:48:09.9248897Z distributed/pipeline/sync/test_microbatch.py::test_batch_atomic PASSED   [ 10%]
2023-01-11T20:48:09.9249228Z distributed/pipeline/sync/test_microbatch.py::test_batch_non_atomic PASSED [ 20%]
2023-01-11T20:48:09.9249556Z distributed/pipeline/sync/test_microbatch.py::test_batch_call PASSED     [ 30%]
2023-01-11T20:48:09.9250062Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_index PASSED [ 40%]
2023-01-11T20:48:09.9250462Z distributed/pipeline/sync/test_microbatch.py::test_batch_setitem_by_slice PASSED [ 50%]
2023-01-11T20:48:09.9250788Z distributed/pipeline/sync/test_microbatch.py::test_check PASSED          [ 60%]
2023-01-11T20:48:09.9251292Z distributed/pipeline/sync/test_microbatch.py::test_gather_tensors PASSED [ 70%]
2023-01-11T20:48:09.9251598Z distributed/pipeline/sync/test_microbatch.py::test_gather_tuples PASSED  [ 80%]
2023-01-11T20:48:09.9251917Z distributed/pipeline/sync/test_microbatch.py::test_scatter_tensor PASSED [ 90%]
2023-01-11T20:48:09.9252254Z distributed/pipeline/sync/test_microbatch.py::test_scatter_multiple_tensors PASSED [100%]
2023-01-11T20:48:09.9252445Z 
2023-01-11T20:48:09.9252554Z ============================== 10 passed in 0.04s ==============================
2023-01-11T20:48:09.9252678Z 
2023-01-11T20:48:09.9252938Z ##[endgroup]
2023-01-11T20:48:09.9253471Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_microbatch (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_microbatch_rh772dbe)
2023-01-11T20:48:09.9253762Z 
2023-01-11T20:48:09.9253988Z Running distributed/pipeline/sync/test_dependency ... [2023-01-11 20:48:09.924554]
2023-01-11T20:48:09.9254444Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_dependency.py', '-v'] ... [2023-01-11 20:48:09.924918]
2023-01-11T20:48:11.5875574Z 
2023-01-11T20:48:11.5876229Z Expand the folded group to see the log file of distributed/pipeline/sync/test_dependency
2023-01-11T20:48:11.5877464Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_dependency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_dependency__5pidrws)
2023-01-11T20:48:11.5878225Z ============================= test session starts ==============================
2023-01-11T20:48:11.5879201Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T20:48:11.5879445Z cachedir: .pytest_cache
2023-01-11T20:48:11.5879890Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T20:48:11.5880359Z torch: 2.0.0a0+git8419ddd
2023-01-11T20:48:11.5880672Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T20:48:11.5881321Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T20:48:11.5881779Z collecting ... collected 6 items
2023-01-11T20:48:11.5882558Z Running 6 items in this shard: test/distributed/pipeline/sync/test_dependency.py::test_fork_join, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad, test/distributed/pipeline/sync/test_dependency.py::test_fork_leak, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad, test/distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad
2023-01-11T20:48:11.5883168Z 
2023-01-11T20:48:11.5883327Z distributed/pipeline/sync/test_dependency.py::test_fork_join SKIPPED     [ 16%]
2023-01-11T20:48:11.5883742Z distributed/pipeline/sync/test_dependency.py::test_fork_join_enable_grad PASSED [ 33%]
2023-01-11T20:48:11.5884306Z distributed/pipeline/sync/test_dependency.py::test_fork_join_no_grad PASSED [ 50%]
2023-01-11T20:48:11.5884821Z distributed/pipeline/sync/test_dependency.py::test_fork_leak PASSED      [ 66%]
2023-01-11T20:48:11.5885160Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_not_requires_grad PASSED [ 83%]
2023-01-11T20:48:11.5885509Z distributed/pipeline/sync/test_dependency.py::test_join_when_fork_requires_grad PASSED [100%]
2023-01-11T20:48:11.5885699Z 
2023-01-11T20:48:11.5885815Z ========================= 5 passed, 1 skipped in 0.03s =========================
2023-01-11T20:48:11.5885963Z 
2023-01-11T20:48:11.5886208Z ##[endgroup]
2023-01-11T20:48:11.5886700Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_dependency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_dependency__5pidrws)
2023-01-11T20:48:11.5886997Z 
2023-01-11T20:48:11.5887210Z Running distributed/pipeline/sync/test_copy ... [2023-01-11 20:48:11.587934]
2023-01-11T20:48:11.5887665Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_copy.py', '-v'] ... [2023-01-11 20:48:11.588257]
2023-01-11T20:48:13.2217156Z 
2023-01-11T20:48:13.2217749Z Expand the folded group to see the log file of distributed/pipeline/sync/test_copy
2023-01-11T20:48:13.2218968Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_copy (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_copy_tsru0eyq)
2023-01-11T20:48:13.2219704Z ============================= test session starts ==============================
2023-01-11T20:48:13.2220416Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T20:48:13.2220663Z cachedir: .pytest_cache
2023-01-11T20:48:13.2221131Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T20:48:13.2221453Z torch: 2.0.0a0+git8419ddd
2023-01-11T20:48:13.2221738Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T20:48:13.2222196Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T20:48:13.2222495Z collecting ... collected 5 items
2023-01-11T20:48:13.2223071Z Running 5 items in this shard: test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda, test/distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors
2023-01-11T20:48:13.2223727Z 
2023-01-11T20:48:13.2223879Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu PASSED    [ 20%]
2023-01-11T20:48:13.2224190Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda SKIPPED  [ 40%]
2023-01-11T20:48:13.2224543Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu SKIPPED  [ 60%]
2023-01-11T20:48:13.2225016Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda SKIPPED [ 80%]
2023-01-11T20:48:13.2225347Z distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors PASSED [100%]
2023-01-11T20:48:13.2225569Z 
2023-01-11T20:48:13.2225680Z ========================= 2 passed, 3 skipped in 0.03s =========================
2023-01-11T20:48:13.2225823Z 
2023-01-11T20:48:13.2226180Z ##[endgroup]
2023-01-11T20:48:13.2226666Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_copy (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_copy_tsru0eyq)
2023-01-11T20:48:13.2227042Z 
2023-01-11T20:48:13.2227243Z Running distributed/pipeline/sync/test_bugs ... [2023-01-11 20:48:13.222072]
2023-01-11T20:48:13.2227695Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_bugs.py', '-v'] ... [2023-01-11 20:48:13.222406]
2023-01-11T20:48:15.0178443Z 
2023-01-11T20:48:15.0179045Z Expand the folded group to see the log file of distributed/pipeline/sync/test_bugs
2023-01-11T20:48:15.0180344Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_bugs (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_bugs_b4r6bh_n)
2023-01-11T20:48:15.0181051Z ============================= test session starts ==============================
2023-01-11T20:48:15.0181790Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T20:48:15.0182220Z cachedir: .pytest_cache
2023-01-11T20:48:15.0182879Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T20:48:15.0183412Z torch: 2.0.0a0+git8419ddd
2023-01-11T20:48:15.0183811Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T20:48:15.0184503Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T20:48:15.0184956Z collecting ... collected 4 items
2023-01-11T20:48:15.0185802Z Running 4 items in this shard: test/distributed/pipeline/sync/test_bugs.py::test_python_autograd_function, test/distributed/pipeline/sync/test_bugs.py::test_exception_no_hang, test/distributed/pipeline/sync/test_bugs.py::test_tuple_wait, test/distributed/pipeline/sync/test_bugs.py::test_parallel_randoms
2023-01-11T20:48:15.0186419Z 
2023-01-11T20:48:15.0186677Z distributed/pipeline/sync/test_bugs.py::test_python_autograd_function PASSED [ 25%]
2023-01-11T20:48:15.0187191Z distributed/pipeline/sync/test_bugs.py::test_exception_no_hang PASSED    [ 50%]
2023-01-11T20:48:15.0187727Z distributed/pipeline/sync/test_bugs.py::test_tuple_wait SKIPPED (2 c...) [ 75%]
2023-01-11T20:48:15.0188242Z distributed/pipeline/sync/test_bugs.py::test_parallel_randoms PASSED     [100%]
2023-01-11T20:48:15.0188496Z 
2023-01-11T20:48:15.0188663Z ========================= 3 passed, 1 skipped in 0.18s =========================
2023-01-11T20:48:15.0188853Z 
2023-01-11T20:48:15.0189205Z ##[endgroup]
2023-01-11T20:48:15.0189905Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_bugs (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_bugs_b4r6bh_n)
2023-01-11T20:48:15.0190309Z 
2023-01-11T20:48:15.0190698Z Running distributed/pipeline/sync/skip/test_verify_skippables ... [2023-01-11 20:48:15.018235]
2023-01-11T20:48:15.0191480Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_verify_skippables.py', '-v'] ... [2023-01-11 20:48:15.018640]
2023-01-11T20:48:16.6820325Z 
2023-01-11T20:48:16.6820963Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_verify_skippables
2023-01-11T20:48:16.6822918Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_verify_skippables (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_verify_skippables_hskg1wsh)
2023-01-11T20:48:16.6823750Z ============================= test session starts ==============================
2023-01-11T20:48:16.6824513Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T20:48:16.6824879Z cachedir: .pytest_cache
2023-01-11T20:48:16.6825399Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T20:48:16.6825994Z torch: 2.0.0a0+git8419ddd
2023-01-11T20:48:16.6826327Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T20:48:16.6826742Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T20:48:16.6827034Z collecting ... collected 9 items
2023-01-11T20:48:16.6828248Z Running 9 items in this shard: test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop, test/distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated
2023-01-11T20:48:16.6829528Z 
2023-01-11T20:48:16.6829699Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_matching PASSED [ 11%]
2023-01-11T20:48:16.6830052Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_not_pop PASSED [ 22%]
2023-01-11T20:48:16.6830402Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_unknown PASSED [ 33%]
2023-01-11T20:48:16.6830732Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_again PASSED [ 44%]
2023-01-11T20:48:16.6831074Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_pop_again PASSED [ 55%]
2023-01-11T20:48:16.6831443Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_different_names PASSED [ 66%]
2023-01-11T20:48:16.6831828Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_stash_pop_together_same_name PASSED [ 77%]
2023-01-11T20:48:16.6832182Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop PASSED [ 88%]
2023-01-11T20:48:16.6832548Z distributed/pipeline/sync/skip/test_verify_skippables.py::test_double_stash_pop_but_isolated PASSED [100%]
2023-01-11T20:48:16.6832753Z 
2023-01-11T20:48:16.6832862Z ============================== 9 passed in 0.03s ===============================
2023-01-11T20:48:16.6832999Z 
2023-01-11T20:48:16.6833227Z ##[endgroup]
2023-01-11T20:48:16.6833777Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_verify_skippables (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_verify_skippables_hskg1wsh)
2023-01-11T20:48:16.6834099Z 
2023-01-11T20:48:16.6834331Z Running distributed/pipeline/sync/skip/test_stash_pop ... [2023-01-11 20:48:16.682363]
2023-01-11T20:48:16.6834800Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_stash_pop.py', '-v'] ... [2023-01-11 20:48:16.682693]
2023-01-11T20:48:18.3439627Z 
2023-01-11T20:48:18.3440307Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_stash_pop
2023-01-11T20:48:18.3441826Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_stash_pop (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_stash_pop_4quirvrh)
2023-01-11T20:48:18.3442510Z ============================= test session starts ==============================
2023-01-11T20:48:18.3443057Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T20:48:18.3443321Z cachedir: .pytest_cache
2023-01-11T20:48:18.3443795Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T20:48:18.3444134Z torch: 2.0.0a0+git8419ddd
2023-01-11T20:48:18.3444441Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T20:48:18.3444852Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T20:48:18.3445198Z collecting ... collected 7 items
2023-01-11T20:48:18.3446011Z Running 7 items in this shard: test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none
2023-01-11T20:48:18.3446621Z 
2023-01-11T20:48:18.3446776Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash PASSED      [ 14%]
2023-01-11T20:48:18.3447095Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop PASSED        [ 28%]
2023-01-11T20:48:18.3447466Z distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use PASSED [ 42%]
2023-01-11T20:48:18.3447795Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared PASSED [ 57%]
2023-01-11T20:48:18.3448179Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared PASSED [ 71%]
2023-01-11T20:48:18.3448517Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed PASSED [ 85%]
2023-01-11T20:48:18.3448830Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none PASSED [100%]
2023-01-11T20:48:18.3449005Z 
2023-01-11T20:48:18.3449113Z ============================== 7 passed in 0.03s ===============================
2023-01-11T20:48:18.3449247Z 
2023-01-11T20:48:18.3449483Z ##[endgroup]
2023-01-11T20:48:18.3449987Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_stash_pop (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_stash_pop_4quirvrh)
2023-01-11T20:48:18.3450286Z 
2023-01-11T20:48:18.3450503Z Running distributed/pipeline/sync/skip/test_leak ... [2023-01-11 20:48:18.344332]
2023-01-11T20:48:18.3451127Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_leak.py', '-v'] ... [2023-01-11 20:48:18.344662]
2023-01-11T20:48:20.1448574Z 
2023-01-11T20:48:20.1449182Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_leak
2023-01-11T20:48:20.1450391Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_leak (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_leak_atur61b6)
2023-01-11T20:48:20.1451008Z ============================= test session starts ==============================
2023-01-11T20:48:20.1451477Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T20:48:20.1451786Z cachedir: .pytest_cache
2023-01-11T20:48:20.1452219Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T20:48:20.1452575Z torch: 2.0.0a0+git8419ddd
2023-01-11T20:48:20.1452865Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T20:48:20.1453444Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T20:48:20.1453725Z collecting ... collected 8 items
2023-01-11T20:48:20.1455383Z Running 8 items in this shard: test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train], test/distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train], test/distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval]
2023-01-11T20:48:20.1456378Z 
2023-01-11T20:48:20.1456746Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-train] PASSED [ 12%]
2023-01-11T20:48:20.1457364Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[always-eval] PASSED [ 25%]
2023-01-11T20:48:20.1457883Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-train] PASSED [ 37%]
2023-01-11T20:48:20.1458324Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[except_last-eval] PASSED [ 50%]
2023-01-11T20:48:20.1458817Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-train] PASSED [ 62%]
2023-01-11T20:48:20.1459391Z distributed/pipeline/sync/skip/test_leak.py::test_delete_portal_tensor[never-eval] PASSED [ 75%]
2023-01-11T20:48:20.1459749Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[train] PASSED [ 87%]
2023-01-11T20:48:20.1460135Z distributed/pipeline/sync/skip/test_leak.py::test_no_portal_without_pipe[eval] PASSED [100%]
2023-01-11T20:48:20.1460323Z 
2023-01-11T20:48:20.1460430Z ============================== 8 passed in 0.19s ===============================
2023-01-11T20:48:20.1460567Z 
2023-01-11T20:48:20.1460856Z ##[endgroup]
2023-01-11T20:48:20.1461352Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_leak (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_leak_atur61b6)
2023-01-11T20:48:20.1461691Z 
2023-01-11T20:48:20.1461916Z Running distributed/pipeline/sync/skip/test_gpipe ... [2023-01-11 20:48:20.145227]
2023-01-11T20:48:20.1462429Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_gpipe.py', '-v'] ... [2023-01-11 20:48:20.145550]
2023-01-11T20:48:21.8487799Z 
2023-01-11T20:48:21.8488377Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_gpipe
2023-01-11T20:48:21.8489458Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_gpipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_gpipe_0trgxl7l)
2023-01-11T20:48:21.8489855Z ============================= test session starts ==============================
2023-01-11T20:48:21.8490339Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T20:48:21.8490610Z cachedir: .pytest_cache
2023-01-11T20:48:21.8491244Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T20:48:21.8491690Z torch: 2.0.0a0+git8419ddd
2023-01-11T20:48:21.8491965Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T20:48:21.8492446Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T20:48:21.8492730Z collecting ... collected 13 items
2023-01-11T20:48:21.8494494Z Running 13 items in this shard: test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1], test/distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip
2023-01-11T20:48:21.8495854Z 
2023-01-11T20:48:21.8496155Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-3] SKIPPED [  7%]
2023-01-11T20:48:21.8496553Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:2] SKIPPED [ 15%]
2023-01-11T20:48:21.8496928Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-2:1] SKIPPED [ 23%]
2023-01-11T20:48:21.8497390Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[never-1:1:1] SKIPPED [ 30%]
2023-01-11T20:48:21.8497787Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-3] SKIPPED [ 38%]
2023-01-11T20:48:21.8498177Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:2] SKIPPED [ 46%]
2023-01-11T20:48:21.8498552Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-2:1] SKIPPED [ 53%]
2023-01-11T20:48:21.8498943Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[always-1:1:1] SKIPPED [ 61%]
2023-01-11T20:48:21.8499341Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-3] SKIPPED [ 69%]
2023-01-11T20:48:21.8499732Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:2] SKIPPED [ 76%]
2023-01-11T20:48:21.8500134Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-2:1] SKIPPED [ 84%]
2023-01-11T20:48:21.8500539Z distributed/pipeline/sync/skip/test_gpipe.py::test_1to3[except_last-1:1:1] SKIPPED [ 92%]
2023-01-11T20:48:21.8500870Z distributed/pipeline/sync/skip/test_gpipe.py::test_none_skip PASSED      [100%]
2023-01-11T20:48:21.8501050Z 
2023-01-11T20:48:21.8501153Z ======================== 1 passed, 12 skipped in 0.09s =========================
2023-01-11T20:48:21.8501297Z 
2023-01-11T20:48:21.8501543Z ##[endgroup]
2023-01-11T20:48:21.8502041Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_gpipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_gpipe_0trgxl7l)
2023-01-11T20:48:21.8502337Z 
2023-01-11T20:48:21.8502541Z Running distributed/fsdp/test_shard_utils ... [2023-01-11 20:48:21.849046]
2023-01-11T20:48:21.8503046Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_shard_utils.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:48:21.849401]
2023-01-11T20:48:23.2721066Z 
2023-01-11T20:48:23.2721666Z Expand the folded group to see the log file of distributed/fsdp/test_shard_utils
2023-01-11T20:48:23.2722601Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_shard_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_shard_utils_mj7_17lc)
2023-01-11T20:48:23.2722883Z 
2023-01-11T20:48:23.2723086Z ##[endgroup]
2023-01-11T20:48:23.2723641Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_shard_utils (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_shard_utils_mj7_17lc)
2023-01-11T20:48:23.2723901Z 
2023-01-11T20:48:23.2725753Z Running distributed/elastic/events/lib_test ... [2023-01-11 20:48:23.272340]
2023-01-11T20:48:23.2728542Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/elastic/events/lib_test.py', '-v'] ... [2023-01-11 20:48:23.272686]
2023-01-11T20:48:24.9680449Z 
2023-01-11T20:48:24.9680913Z Expand the folded group to see the log file of distributed/elastic/events/lib_test
2023-01-11T20:48:24.9681640Z ##[group]PRINTING LOG FILE of distributed/elastic/events/lib_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-events-lib_test_vgwbb37m)
2023-01-11T20:48:24.9682278Z ============================= test session starts ==============================
2023-01-11T20:48:24.9682745Z platform linux -- Python 3.7.15, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T20:48:24.9683004Z cachedir: .pytest_cache
2023-01-11T20:48:24.9683420Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T20:48:24.9683795Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T20:48:24.9684353Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T20:48:24.9684650Z collecting ... collected 8 items
2023-01-11T20:48:24.9685743Z Running 8 items in this shard: test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_created, test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser, test/distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str
2023-01-11T20:48:24.9686580Z 
2023-01-11T20:48:24.9686745Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_created PASSED [ 12%]
2023-01-11T20:48:24.9687083Z distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser PASSED [ 25%]
2023-01-11T20:48:24.9687515Z distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger PASSED [ 37%]
2023-01-11T20:48:24.9687932Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event PASSED [ 50%]
2023-01-11T20:48:24.9688465Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest PASSED [ 62%]
2023-01-11T20:48:24.9688886Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created PASSED [ 75%]
2023-01-11T20:48:24.9689250Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize PASSED [ 87%]
2023-01-11T20:48:24.9689657Z distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str PASSED [100%]
2023-01-11T20:48:24.9689844Z 
2023-01-11T20:48:24.9689955Z ============================== 8 passed in 1.14s ===============================
2023-01-11T20:48:24.9690091Z 
2023-01-11T20:48:24.9690367Z ##[endgroup]
2023-01-11T20:48:24.9691034Z FINISHED PRINTING LOG FILE of distributed/elastic/events/lib_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-events-lib_test_vgwbb37m)
2023-01-11T20:48:24.9691305Z 
2023-01-11T20:48:24.9691518Z Running distributed/_shard/test_sharder ... [2023-01-11 20:48:24.968343]
2023-01-11T20:48:24.9692069Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:48:24.968684]
2023-01-11T20:48:26.3667366Z 
2023-01-11T20:48:26.3668104Z Expand the folded group to see the log file of distributed/_shard/test_sharder
2023-01-11T20:48:26.3669267Z ##[group]PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_sharder_9resdtrf)
2023-01-11T20:48:26.3670070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpys5xhh73
2023-01-11T20:48:26.3670779Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpys5xhh73/_remote_module_non_scriptable.py
2023-01-11T20:48:26.3671148Z 
2023-01-11T20:48:26.3671460Z ##[endgroup]
2023-01-11T20:48:26.3672292Z FINISHED PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_sharder_9resdtrf)
2023-01-11T20:48:26.3672997Z 
2023-01-11T20:48:26.3673415Z Running distributed/_shard/sharded_tensor/ops/test_math_ops ... [2023-01-11 20:48:26.367129]
2023-01-11T20:48:26.3677715Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_math_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:48:26.367543]
2023-01-11T20:48:27.7640876Z 
2023-01-11T20:48:27.7641472Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_math_ops
2023-01-11T20:48:27.7642860Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_math_ops_pb60qiti)
2023-01-11T20:48:27.7643407Z 
2023-01-11T20:48:27.7643760Z ##[endgroup]
2023-01-11T20:48:27.7644965Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_math_ops_pb60qiti)
2023-01-11T20:48:27.7645467Z 
2023-01-11T20:48:27.7646339Z Running distributed/test_c10d_spawn_ucc ... [2023-01-11 20:48:27.764450]
2023-01-11T20:48:27.7651291Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_ucc.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:48:27.764865]
2023-01-11T20:48:44.6635232Z 
2023-01-11T20:48:44.6635905Z Expand the folded group to see the log file of distributed/test_c10d_spawn_ucc
2023-01-11T20:48:44.6636932Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_ucc (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_ucc_gyiv6zsi)
2023-01-11T20:48:44.6637624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_m9yp_w
2023-01-11T20:48:44.6638240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_m9yp_w/_remote_module_non_scriptable.py
2023-01-11T20:48:44.6639228Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupShareTensorTest testMethod=test_shared_allgather_ucc>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_ucc>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_ucc>]>
2023-01-11T20:48:44.6640062Z test_shared_allgather_ucc (__main__.ProcessGroupShareTensorTest)
2023-01-11T20:48:44.6640544Z test_shared_allreduce_ucc (__main__.ProcessGroupShareTensorTest)
2023-01-11T20:48:44.6641021Z test_shared_broadcast_ucc (__main__.ProcessGroupShareTensorTest)
2023-01-11T20:48:44.6641452Z <unittest.suite.TestSuite tests=[]>
2023-01-11T20:48:44.6641825Z <unittest.suite.TestSuite tests=[]>
2023-01-11T20:48:44.6642946Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsUcc testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_reduce>]>
2023-01-11T20:48:44.6644138Z test_all_gather (__main__.TestDistributedNNFunctionsUcc)
2023-01-11T20:48:44.6644560Z test_all_to_all (__main__.TestDistributedNNFunctionsUcc)
2023-01-11T20:48:44.6645010Z test_all_to_all_single (__main__.TestDistributedNNFunctionsUcc)
2023-01-11T20:48:44.6645457Z test_allreduce (__main__.TestDistributedNNFunctionsUcc)
2023-01-11T20:48:44.6645888Z test_broadcast (__main__.TestDistributedNNFunctionsUcc)
2023-01-11T20:48:44.6646297Z test_reduce (__main__.TestDistributedNNFunctionsUcc)
2023-01-11T20:48:44.6647099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:44.6647623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:44.6648299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:44.6648847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:44.6649643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8dbq1lma
2023-01-11T20:48:44.6650280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8dbq1lma/_remote_module_non_scriptable.py
2023-01-11T20:48:44.6651173Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2023-01-11T20:48:44.6651486Z 
2023-01-11T20:48:44.6651588Z Running tests...
2023-01-11T20:48:44.6652052Z ----------------------------------------------------------------------
2023-01-11T20:48:44.6652545Z   test_shared_allgather_ucc (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2023-01-11T20:48:44.6652873Z 
2023-01-11T20:48:44.6653144Z ----------------------------------------------------------------------
2023-01-11T20:48:44.6653492Z Ran 1 test in 0.001s
2023-01-11T20:48:44.6653636Z 
2023-01-11T20:48:44.6653739Z OK (skipped=1)
2023-01-11T20:48:44.6653910Z 
2023-01-11T20:48:44.6654208Z Generating XML reports...
2023-01-11T20:48:44.6655084Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-ProcessGroupShareTensorTest-20230111204830.xml
2023-01-11T20:48:44.6656179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:44.6656605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:44.6657060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:44.6657453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:44.6657799Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp982_w1k
2023-01-11T20:48:44.6658249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp982_w1k/_remote_module_non_scriptable.py
2023-01-11T20:48:44.6658763Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2023-01-11T20:48:44.6658964Z 
2023-01-11T20:48:44.6659037Z Running tests...
2023-01-11T20:48:44.6659340Z ----------------------------------------------------------------------
2023-01-11T20:48:44.6659749Z   test_shared_allreduce_ucc (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2023-01-11T20:48:44.6659966Z 
2023-01-11T20:48:44.6660180Z ----------------------------------------------------------------------
2023-01-11T20:48:44.6660457Z Ran 1 test in 0.001s
2023-01-11T20:48:44.6660567Z 
2023-01-11T20:48:44.6660625Z OK (skipped=1)
2023-01-11T20:48:44.6660728Z 
2023-01-11T20:48:44.6660831Z Generating XML reports...
2023-01-11T20:48:44.6661566Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-ProcessGroupShareTensorTest-20230111204832.xml
2023-01-11T20:48:44.6662357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:44.6662819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:44.6663467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:44.6663958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:44.6664397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptqw74e2j
2023-01-11T20:48:44.6664798Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptqw74e2j/_remote_module_non_scriptable.py
2023-01-11T20:48:44.6665316Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2023-01-11T20:48:44.6665510Z 
2023-01-11T20:48:44.6665580Z Running tests...
2023-01-11T20:48:44.6665930Z ----------------------------------------------------------------------
2023-01-11T20:48:44.6666286Z   test_shared_broadcast_ucc (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2023-01-11T20:48:44.6666674Z 
2023-01-11T20:48:44.6666874Z ----------------------------------------------------------------------
2023-01-11T20:48:44.6667113Z Ran 1 test in 0.001s
2023-01-11T20:48:44.6667213Z 
2023-01-11T20:48:44.6667300Z OK (skipped=1)
2023-01-11T20:48:44.6667452Z 
2023-01-11T20:48:44.6667539Z Generating XML reports...
2023-01-11T20:48:44.6668015Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-ProcessGroupShareTensorTest-20230111204833.xml
2023-01-11T20:48:44.6668623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:44.6669013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:44.6669447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:44.6669852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:44.6670233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2tqeller
2023-01-11T20:48:44.6670691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2tqeller/_remote_module_non_scriptable.py
2023-01-11T20:48:44.6671149Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2023-01-11T20:48:44.6671405Z 
2023-01-11T20:48:44.6671464Z Running tests...
2023-01-11T20:48:44.6671766Z ----------------------------------------------------------------------
2023-01-11T20:48:44.6672191Z   test_all_gather (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s)
2023-01-11T20:48:44.6672413Z 
2023-01-11T20:48:44.6672609Z ----------------------------------------------------------------------
2023-01-11T20:48:44.6672890Z Ran 1 test in 0.001s
2023-01-11T20:48:44.6673003Z 
2023-01-11T20:48:44.6673078Z OK (skipped=1)
2023-01-11T20:48:44.6673183Z 
2023-01-11T20:48:44.6673268Z Generating XML reports...
2023-01-11T20:48:44.6673796Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111204835.xml
2023-01-11T20:48:44.6674394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:44.6674728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:44.6675218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:44.6675549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:44.6675933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxpnel18h
2023-01-11T20:48:44.6676350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxpnel18h/_remote_module_non_scriptable.py
2023-01-11T20:48:44.6676868Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2023-01-11T20:48:44.6677057Z 
2023-01-11T20:48:44.6677132Z Running tests...
2023-01-11T20:48:44.6677465Z ----------------------------------------------------------------------
2023-01-11T20:48:44.6677853Z   test_all_to_all (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s)
2023-01-11T20:48:44.6678068Z 
2023-01-11T20:48:44.6678308Z ----------------------------------------------------------------------
2023-01-11T20:48:44.6678548Z Ran 1 test in 0.000s
2023-01-11T20:48:44.6678660Z 
2023-01-11T20:48:44.6678730Z OK (skipped=1)
2023-01-11T20:48:44.6678837Z 
2023-01-11T20:48:44.6678920Z Generating XML reports...
2023-01-11T20:48:44.6679451Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111204837.xml
2023-01-11T20:48:44.6680049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:44.6680386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:44.6680842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:44.6681182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:44.6681578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34qbuz4k
2023-01-11T20:48:44.6681976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34qbuz4k/_remote_module_non_scriptable.py
2023-01-11T20:48:44.6682416Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2023-01-11T20:48:44.6682612Z 
2023-01-11T20:48:44.6682684Z Running tests...
2023-01-11T20:48:44.6682985Z ----------------------------------------------------------------------
2023-01-11T20:48:44.6683339Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s)
2023-01-11T20:48:44.6683621Z 
2023-01-11T20:48:44.6683858Z ----------------------------------------------------------------------
2023-01-11T20:48:44.6684094Z Ran 1 test in 0.001s
2023-01-11T20:48:44.6684207Z 
2023-01-11T20:48:44.6684278Z OK (skipped=1)
2023-01-11T20:48:44.6684384Z 
2023-01-11T20:48:44.6684454Z Generating XML reports...
2023-01-11T20:48:44.6684929Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111204839.xml
2023-01-11T20:48:44.6685481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:44.6685841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:44.6686257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:44.6686605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:44.6686952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3qv50yqd
2023-01-11T20:48:44.6687416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3qv50yqd/_remote_module_non_scriptable.py
2023-01-11T20:48:44.6687859Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2023-01-11T20:48:44.6688050Z 
2023-01-11T20:48:44.6688122Z Running tests...
2023-01-11T20:48:44.6688418Z ----------------------------------------------------------------------
2023-01-11T20:48:44.6688771Z   test_allreduce (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s)
2023-01-11T20:48:44.6688990Z 
2023-01-11T20:48:44.6689182Z ----------------------------------------------------------------------
2023-01-11T20:48:44.6689416Z Ran 1 test in 0.001s
2023-01-11T20:48:44.6689525Z 
2023-01-11T20:48:44.6689596Z OK (skipped=1)
2023-01-11T20:48:44.6689701Z 
2023-01-11T20:48:44.6689771Z Generating XML reports...
2023-01-11T20:48:44.6690250Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111204840.xml
2023-01-11T20:48:44.6690801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:44.6691228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:44.6691639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:44.6691982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:44.6692326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplimho7h_
2023-01-11T20:48:44.6692712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplimho7h_/_remote_module_non_scriptable.py
2023-01-11T20:48:44.6693157Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2023-01-11T20:48:44.6693353Z 
2023-01-11T20:48:44.6693498Z Running tests...
2023-01-11T20:48:44.6693802Z ----------------------------------------------------------------------
2023-01-11T20:48:44.6694153Z   test_broadcast (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s)
2023-01-11T20:48:44.6694440Z 
2023-01-11T20:48:44.6694634Z ----------------------------------------------------------------------
2023-01-11T20:48:44.6694873Z Ran 1 test in 0.000s
2023-01-11T20:48:44.6694988Z 
2023-01-11T20:48:44.6695046Z OK (skipped=1)
2023-01-11T20:48:44.6695154Z 
2023-01-11T20:48:44.6695237Z Generating XML reports...
2023-01-11T20:48:44.6695710Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111204842.xml
2023-01-11T20:48:44.6696335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:48:44.6696649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:48:44.6697136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:48:44.6697481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:48:44.6697824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbf78te17
2023-01-11T20:48:44.6698209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbf78te17/_remote_module_non_scriptable.py
2023-01-11T20:48:44.6698659Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2023-01-11T20:48:44.6698854Z 
2023-01-11T20:48:44.6698927Z Running tests...
2023-01-11T20:48:44.6699215Z ----------------------------------------------------------------------
2023-01-11T20:48:44.6699573Z   test_reduce (__main__.TestDistributedNNFunctionsUcc) ... skip: c10d was not compiled with the UCC backend (0.000s)
2023-01-11T20:48:44.6699787Z 
2023-01-11T20:48:44.6699982Z ----------------------------------------------------------------------
2023-01-11T20:48:44.6700221Z Ran 1 test in 0.000s
2023-01-11T20:48:44.6700320Z 
2023-01-11T20:48:44.6700391Z OK (skipped=1)
2023-01-11T20:48:44.6700493Z 
2023-01-11T20:48:44.6700575Z Generating XML reports...
2023-01-11T20:48:44.6701048Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111204844.xml
2023-01-11T20:48:44.6701329Z 
2023-01-11T20:48:44.6701623Z ##[endgroup]
2023-01-11T20:48:44.6702053Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_ucc (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_ucc_gyiv6zsi)
2023-01-11T20:48:44.6702295Z 
2023-01-11T20:48:44.6702517Z Running distributed/_shard/sharded_tensor/ops/test_softmax ... [2023-01-11 20:48:44.663913]
2023-01-11T20:48:44.6703070Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_softmax.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:48:44.664366]
2023-01-11T20:48:46.0907138Z 
2023-01-11T20:48:46.0907772Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_softmax
2023-01-11T20:48:46.0908742Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_softmax (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_softmax_ocvj6s1z)
2023-01-11T20:48:46.0909249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt7jtri5f
2023-01-11T20:48:46.0909648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt7jtri5f/_remote_module_non_scriptable.py
2023-01-11T20:48:46.0910240Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax
2023-01-11T20:48:46.0910458Z 
2023-01-11T20:48:46.0910535Z Running tests...
2023-01-11T20:48:46.0910870Z ----------------------------------------------------------------------
2023-01-11T20:48:46.0911265Z   test_sharded_softmax_basic (__main__.TestShardedSoftmax) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:48:46.0911969Z   test_sharded_softmax_on_sharding_dim (__main__.TestShardedSoftmax) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:48:46.0912243Z 
2023-01-11T20:48:46.0912494Z ----------------------------------------------------------------------
2023-01-11T20:48:46.0912742Z Ran 2 tests in 0.001s
2023-01-11T20:48:46.0912879Z 
2023-01-11T20:48:46.0912955Z OK (skipped=2)
2023-01-11T20:48:46.0913063Z 
2023-01-11T20:48:46.0913147Z Generating XML reports...
2023-01-11T20:48:46.0913637Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20230111204845.xml
2023-01-11T20:48:46.0913979Z 
2023-01-11T20:48:46.0914290Z ##[endgroup]
2023-01-11T20:48:46.0914949Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_softmax (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_softmax_ocvj6s1z)
2023-01-11T20:48:46.0915245Z 
2023-01-11T20:48:46.0915492Z Running distributed/_shard/sharded_tensor/ops/test_embedding_bag ... [2023-01-11 20:48:46.090978]
2023-01-11T20:48:46.0916067Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:48:46.091324]
2023-01-11T20:48:47.5042914Z 
2023-01-11T20:48:47.5043579Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_embedding_bag
2023-01-11T20:48:47.5044597Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding_bag (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_bag_38ggm8cq)
2023-01-11T20:48:47.5045285Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag
2023-01-11T20:48:47.5045512Z 
2023-01-11T20:48:47.5045587Z Running tests...
2023-01-11T20:48:47.5045918Z ----------------------------------------------------------------------
2023-01-11T20:48:47.5046377Z   test_sharded_embedding_bag_colwise (__main__.TestShardedEmbeddingBag) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:48:47.5046921Z   test_sharded_embedding_bag_rowwise (__main__.TestShardedEmbeddingBag) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:48:47.5047149Z 
2023-01-11T20:48:47.5047351Z ----------------------------------------------------------------------
2023-01-11T20:48:47.5047589Z Ran 2 tests in 0.001s
2023-01-11T20:48:47.5047704Z 
2023-01-11T20:48:47.5047776Z OK (skipped=2)
2023-01-11T20:48:47.5047871Z 
2023-01-11T20:48:47.5047956Z Generating XML reports...
2023-01-11T20:48:47.5048472Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20230111204847.xml
2023-01-11T20:48:47.5048760Z 
2023-01-11T20:48:47.5048991Z ##[endgroup]
2023-01-11T20:48:47.5049545Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding_bag (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_bag_38ggm8cq)
2023-01-11T20:48:47.5049915Z 
2023-01-11T20:48:47.5050158Z Running distributed/_shard/sharded_tensor/ops/test_binary_cmp ... [2023-01-11 20:48:47.504555]
2023-01-11T20:48:47.5050718Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:48:47.504887]
2023-01-11T20:48:48.9143923Z 
2023-01-11T20:48:48.9144769Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_binary_cmp
2023-01-11T20:48:48.9145713Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_binary_cmp (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_binary_cmp_yh_8y209)
2023-01-11T20:48:48.9146400Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp
2023-01-11T20:48:48.9146846Z 
2023-01-11T20:48:48.9146927Z Running tests...
2023-01-11T20:48:48.9147258Z ----------------------------------------------------------------------
2023-01-11T20:48:48.9147567Z   test_torch_allclose (__main__.TestShardedTensorBinaryOps)
2023-01-11T20:48:48.9147962Z Test torch.allclose(ShardedTensor, ShardedTensor) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:48:48.9148508Z   test_torch_allclose_tensor_specs (__main__.TestShardedTensorBinaryOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:48:48.9148908Z   test_torch_equal (__main__.TestShardedTensorBinaryOps)
2023-01-11T20:48:48.9149247Z Test torch.equal(ShardedTensor, ShardedTensor) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:48:48.9149747Z   test_torch_equal_tensor_specs (__main__.TestShardedTensorBinaryOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:48:48.9149980Z 
2023-01-11T20:48:48.9150188Z ----------------------------------------------------------------------
2023-01-11T20:48:48.9150451Z Ran 4 tests in 0.002s
2023-01-11T20:48:48.9150567Z 
2023-01-11T20:48:48.9150639Z OK (skipped=4)
2023-01-11T20:48:48.9150749Z 
2023-01-11T20:48:48.9150834Z Generating XML reports...
2023-01-11T20:48:48.9151375Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20230111204848.xml
2023-01-11T20:48:48.9151708Z 
2023-01-11T20:48:48.9151957Z ##[endgroup]
2023-01-11T20:48:48.9152473Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_binary_cmp (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_binary_cmp_yh_8y209)
2023-01-11T20:48:48.9152766Z 
2023-01-11T20:48:48.9152988Z Running distributed/algorithms/ddp_comm_hooks/test_ddp_hooks ... [2023-01-11 20:48:48.914664]
2023-01-11T20:48:48.9153561Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:48:48.915005]
2023-01-11T20:48:50.3097286Z 
2023-01-11T20:48:50.3097926Z Expand the folded group to see the log file of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2023-01-11T20:48:50.3098918Z ##[group]PRINTING LOG FILE of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-ddp_comm_hooks-test_ddp_hooks_w272w46h)
2023-01-11T20:48:50.3099623Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks
2023-01-11T20:48:50.3099837Z 
2023-01-11T20:48:50.3099913Z Running tests...
2023-01-11T20:48:50.3100219Z ----------------------------------------------------------------------
2023-01-11T20:48:50.3100558Z   test_ddp_comm_hook_allreduce_hook (__main__.DistributedDataParallelCommHookTest)
2023-01-11T20:48:50.3100974Z This unit test verifies the ``allreduce`` hook registered case gives same result ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:48:50.3101379Z   test_ddp_comm_hook_fp16compress_hook (__main__.DistributedDataParallelCommHookTest)
2023-01-11T20:48:50.3101811Z This unit test verifies the ``fp16 compress`` hook registered case ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:48:50.3102258Z   test_ddp_comm_hook_noop_hook (__main__.DistributedDataParallelCommHookTest)
2023-01-11T20:48:50.3102792Z This unit test verifies the ``noop`` hook registered case and a subsequent allreduce ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:48:50.3103194Z   test_ddp_comm_hook_quantize_per_channel_hook (__main__.DistributedDataParallelCommHookTest)
2023-01-11T20:48:50.3103598Z This unit test verifies the ``quantize per channel`` hook registered case ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:48:50.3104161Z   test_ddp_comm_hook_quantize_per_tensor_hook (__main__.DistributedDataParallelCommHookTest)
2023-01-11T20:48:50.3104547Z This unit test verifies the ``quantize per tensor`` hook registered case ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:48:50.3104971Z   test_is_last_hook (__main__.DistributedDataParallelCommHookTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:48:50.3105203Z 
2023-01-11T20:48:50.3105414Z ----------------------------------------------------------------------
2023-01-11T20:48:50.3105655Z Ran 6 tests in 0.003s
2023-01-11T20:48:50.3105794Z 
2023-01-11T20:48:50.3105899Z OK (skipped=6)
2023-01-11T20:48:50.3106007Z 
2023-01-11T20:48:50.3106092Z Generating XML reports...
2023-01-11T20:48:50.3106650Z Generated XML report: test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20230111204850.xml
2023-01-11T20:48:50.3106979Z 
2023-01-11T20:48:50.3107207Z ##[endgroup]
2023-01-11T20:48:50.3107812Z FINISHED PRINTING LOG FILE of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-ddp_comm_hooks-test_ddp_hooks_w272w46h)
2023-01-11T20:48:50.3108113Z 
2023-01-11T20:48:50.3108323Z Running distributed/_tools/test_memory_tracker ... [2023-01-11 20:48:50.310074]
2023-01-11T20:48:50.3108849Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tools/test_memory_tracker.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:48:50.310458]
2023-01-11T20:48:52.0229045Z 
2023-01-11T20:48:52.0229681Z Expand the folded group to see the log file of distributed/_tools/test_memory_tracker
2023-01-11T20:48:52.0230543Z ##[group]PRINTING LOG FILE of distributed/_tools/test_memory_tracker (/var/lib/jenkins/workspace/test/test-reports/distributed-_tools-test_memory_tracker_m25ixje8)
2023-01-11T20:48:52.0231560Z Test results will be stored in test-reports/python-unittest/distributed._tools.test_memory_tracker
2023-01-11T20:48:52.0231772Z 
2023-01-11T20:48:52.0231847Z Running tests...
2023-01-11T20:48:52.0232161Z ----------------------------------------------------------------------
2023-01-11T20:48:52.0232443Z   test_local_model (__main__.TestMemoryTracker)
2023-01-11T20:48:52.0232805Z Minimal test case to check the memory tracker can collect the expected ... skip: no cuda (0.002s)
2023-01-11T20:48:52.0232994Z 
2023-01-11T20:48:52.0233245Z ----------------------------------------------------------------------
2023-01-11T20:48:52.0233594Z Ran 1 test in 0.002s
2023-01-11T20:48:52.0233712Z 
2023-01-11T20:48:52.0233772Z OK (skipped=1)
2023-01-11T20:48:52.0233928Z 
2023-01-11T20:48:52.0234020Z Generating XML reports...
2023-01-11T20:48:52.0234484Z Generated XML report: test-reports/python-unittest/distributed._tools.test_memory_tracker/TEST-TestMemoryTracker-20230111204851.xml
2023-01-11T20:48:52.0234745Z 
2023-01-11T20:48:52.0234959Z ##[endgroup]
2023-01-11T20:48:52.0235421Z FINISHED PRINTING LOG FILE of distributed/_tools/test_memory_tracker (/var/lib/jenkins/workspace/test/test-reports/distributed-_tools-test_memory_tracker_m25ixje8)
2023-01-11T20:48:52.0235681Z 
2023-01-11T20:48:52.0236021Z Running distributed/_shard/sharded_tensor/test_sharded_tensor_reshard ... [2023-01-11 20:48:52.023184]
2023-01-11T20:48:52.0237050Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:48:52.023532]
2023-01-11T20:48:53.4446940Z 
2023-01-11T20:48:53.4447435Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard
2023-01-11T20:48:53.4448281Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_reshard_jniwezsd)
2023-01-11T20:48:53.4449047Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard
2023-01-11T20:48:53.4449543Z 
2023-01-11T20:48:53.4449637Z Running tests...
2023-01-11T20:48:53.4449949Z ----------------------------------------------------------------------
2023-01-11T20:48:53.4450466Z   test_sharded_tensor_reshard (__main__.TestReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:48:53.4451161Z   test_sharded_tensor_reshard_errors (__main__.TestReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:48:53.4451384Z 
2023-01-11T20:48:53.4451595Z ----------------------------------------------------------------------
2023-01-11T20:48:53.4451828Z Ran 2 tests in 0.002s
2023-01-11T20:48:53.4451942Z 
2023-01-11T20:48:53.4452013Z OK (skipped=2)
2023-01-11T20:48:53.4452120Z 
2023-01-11T20:48:53.4452205Z Generating XML reports...
2023-01-11T20:48:53.4452800Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20230111204853.xml
2023-01-11T20:48:53.4453077Z 
2023-01-11T20:48:53.4453361Z ##[endgroup]
2023-01-11T20:48:53.4453902Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_sharded_tensor_reshard (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_sharded_tensor_reshard_jniwezsd)
2023-01-11T20:48:53.4454286Z 
2023-01-11T20:48:53.4454554Z Running distributed/_shard/sharded_tensor/test_megatron_prototype ... [2023-01-11 20:48:53.444931]
2023-01-11T20:48:53.4455229Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_megatron_prototype.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:48:53.445298]
2023-01-11T20:48:54.8799930Z 
2023-01-11T20:48:54.8800398Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_megatron_prototype
2023-01-11T20:48:54.8801243Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_megatron_prototype (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_megatron_prototype_enihoeee)
2023-01-11T20:48:54.8802004Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype
2023-01-11T20:48:54.8802232Z 
2023-01-11T20:48:54.8802306Z Running tests...
2023-01-11T20:48:54.8802661Z ----------------------------------------------------------------------
2023-01-11T20:48:54.8803045Z   test_megatron_two_layer_prototype (__main__.TestShardedTensorMegatronLinear) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T20:48:54.8803346Z 
2023-01-11T20:48:54.8803585Z ----------------------------------------------------------------------
2023-01-11T20:48:54.8803901Z Ran 1 test in 0.002s
2023-01-11T20:48:54.8804062Z 
2023-01-11T20:48:54.8804137Z OK (skipped=1)
2023-01-11T20:48:54.8804231Z 
2023-01-11T20:48:54.8804314Z Generating XML reports...
2023-01-11T20:48:54.8804923Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20230111204854.xml
2023-01-11T20:48:54.8805243Z 
2023-01-11T20:48:54.8805468Z ##[endgroup]
2023-01-11T20:48:54.8806026Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_megatron_prototype (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_megatron_prototype_enihoeee)
2023-01-11T20:48:54.8806371Z 
2023-01-11T20:48:54.8806608Z Running distributed/_shard/sharded_tensor/ops/test_embedding ... [2023-01-11 20:48:54.880236]
2023-01-11T20:48:54.8807707Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:48:54.880586]
2023-01-11T20:48:56.3052101Z 
2023-01-11T20:48:56.3052689Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_embedding
2023-01-11T20:48:56.3053683Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_vr3ivncc)
2023-01-11T20:48:56.3054584Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding
2023-01-11T20:48:56.3054841Z 
2023-01-11T20:48:56.3054916Z Running tests...
2023-01-11T20:48:56.3055217Z ----------------------------------------------------------------------
2023-01-11T20:48:56.3055626Z   test_sharded_embedding_colwise (__main__.TestShardedEmbedding) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:48:56.3056088Z   test_sharded_embedding_rowwise (__main__.TestShardedEmbedding) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:48:56.3056377Z 
2023-01-11T20:48:56.3056577Z ----------------------------------------------------------------------
2023-01-11T20:48:56.3056805Z Ran 2 tests in 0.002s
2023-01-11T20:48:56.3056919Z 
2023-01-11T20:48:56.3056991Z OK (skipped=2)
2023-01-11T20:48:56.3057193Z 
2023-01-11T20:48:56.3057282Z Generating XML reports...
2023-01-11T20:48:56.3057789Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20230111204856.xml
2023-01-11T20:48:56.3058058Z 
2023-01-11T20:48:56.3058327Z ##[endgroup]
2023-01-11T20:48:56.3058887Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_vr3ivncc)
2023-01-11T20:48:56.3059234Z 
2023-01-11T20:48:56.3059506Z Running distributed/_shard/sharded_tensor/ops/test_chunk ... [2023-01-11 20:48:56.305444]
2023-01-11T20:48:56.3060046Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_chunk.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:48:56.305796]
2023-01-11T20:48:57.7335543Z 
2023-01-11T20:48:57.7336256Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_chunk
2023-01-11T20:48:57.7337168Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_chunk (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_chunk_j6ukyxw0)
2023-01-11T20:48:57.7337806Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk
2023-01-11T20:48:57.7338048Z 
2023-01-11T20:48:57.7338126Z Running tests...
2023-01-11T20:48:57.7338433Z ----------------------------------------------------------------------
2023-01-11T20:48:57.7338799Z   test_sharded_chunk (__main__.TestShardedTensorChunkOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:48:57.7339200Z   test_sharded_chunk_error (__main__.TestShardedTensorChunkOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:48:57.7339421Z 
2023-01-11T20:48:57.7339670Z ----------------------------------------------------------------------
2023-01-11T20:48:57.7339948Z Ran 2 tests in 0.001s
2023-01-11T20:48:57.7340064Z 
2023-01-11T20:48:57.7340156Z OK (skipped=2)
2023-01-11T20:48:57.7340272Z 
2023-01-11T20:48:57.7340343Z Generating XML reports...
2023-01-11T20:48:57.7340850Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20230111204857.xml
2023-01-11T20:48:57.7341139Z 
2023-01-11T20:48:57.7341358Z ##[endgroup]
2023-01-11T20:48:57.7341841Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_chunk (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_chunk_j6ukyxw0)
2023-01-11T20:48:57.7342124Z 
2023-01-11T20:48:57.7342354Z Running distributed/_shard/sharded_optim/test_sharded_optim ... [2023-01-11 20:48:57.733855]
2023-01-11T20:48:57.7343609Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:48:57.734183]
2023-01-11T20:48:59.1675741Z 
2023-01-11T20:48:59.1676330Z Expand the folded group to see the log file of distributed/_shard/sharded_optim/test_sharded_optim
2023-01-11T20:48:59.1677517Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_optim/test_sharded_optim (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_optim-test_sharded_optim_nomj24it)
2023-01-11T20:48:59.1678457Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim
2023-01-11T20:48:59.1678796Z 
2023-01-11T20:48:59.1678934Z Running tests...
2023-01-11T20:48:59.1679532Z ----------------------------------------------------------------------
2023-01-11T20:48:59.1679962Z   test_named_params_with_sharded_tensor (__main__.TestShardedOptimizer) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:48:59.1680407Z   test_sharded_optim (__main__.TestShardedOptimizer) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:48:59.1680934Z 
2023-01-11T20:48:59.1681206Z ----------------------------------------------------------------------
2023-01-11T20:48:59.1681573Z Ran 2 tests in 0.002s
2023-01-11T20:48:59.1681778Z 
2023-01-11T20:48:59.1681909Z OK (skipped=2)
2023-01-11T20:48:59.1682101Z 
2023-01-11T20:48:59.1682274Z Generating XML reports...
2023-01-11T20:48:59.1682802Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20230111204858.xml
2023-01-11T20:48:59.1683068Z 
2023-01-11T20:48:59.1683328Z ##[endgroup]
2023-01-11T20:48:59.1683881Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_optim/test_sharded_optim (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_optim-test_sharded_optim_nomj24it)
2023-01-11T20:48:59.1684194Z 
2023-01-11T20:48:59.1684415Z Running distributed/test_c10d_spawn_nccl ... [2023-01-11 20:48:59.167814]
2023-01-11T20:48:59.1684942Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:48:59.168261]
2023-01-11T20:49:23.2903987Z 
2023-01-11T20:49:23.2904700Z Expand the folded group to see the log file of distributed/test_c10d_spawn_nccl
2023-01-11T20:49:23.2905813Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_nccl_n_tuzf3w)
2023-01-11T20:49:23.2906585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3d0nkio
2023-01-11T20:49:23.2907304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3d0nkio/_remote_module_non_scriptable.py
2023-01-11T20:49:23.2908582Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupShareTensorTest testMethod=test_shared_allgather_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_allreduce_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_broadcast_nccl>, <__main__.ProcessGroupShareTensorTest testMethod=test_shared_reduce_nccl>]>
2023-01-11T20:49:23.2912476Z test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest)
2023-01-11T20:49:23.2912841Z test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest)
2023-01-11T20:49:23.2913321Z test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest)
2023-01-11T20:49:23.2913631Z test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest)
2023-01-11T20:49:23.2913946Z <unittest.suite.TestSuite tests=[]>
2023-01-11T20:49:23.2914187Z <unittest.suite.TestSuite tests=[]>
2023-01-11T20:49:23.2915288Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather_base>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter_non_contiguous>]>
2023-01-11T20:49:23.2916682Z test_all_gather (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T20:49:23.2917055Z test_all_gather_base (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T20:49:23.2917339Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T20:49:23.2917683Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T20:49:23.2917987Z test_allreduce (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T20:49:23.2918265Z test_broadcast (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T20:49:23.2918607Z test_reduce (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T20:49:23.2918903Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T20:49:23.2919350Z test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T20:49:23.2919977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:49:23.2920317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:49:23.2920806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:49:23.2921143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:49:23.2921539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo4gfsg93
2023-01-11T20:49:23.2921944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo4gfsg93/_remote_module_non_scriptable.py
2023-01-11T20:49:23.2922441Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T20:49:23.2922643Z 
2023-01-11T20:49:23.2922703Z Running tests...
2023-01-11T20:49:23.2923056Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2923421Z   test_shared_allgather_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2023-01-11T20:49:23.2923652Z 
2023-01-11T20:49:23.2923884Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2924108Z Ran 1 test in 0.001s
2023-01-11T20:49:23.2924219Z 
2023-01-11T20:49:23.2924291Z OK (skipped=1)
2023-01-11T20:49:23.2924409Z 
2023-01-11T20:49:23.2924530Z Generating XML reports...
2023-01-11T20:49:23.2924997Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20230111204902.xml
2023-01-11T20:49:23.2925598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:49:23.2925945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:49:23.2926403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:49:23.2926786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:49:23.2927133Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe8vhwi62
2023-01-11T20:49:23.2927582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe8vhwi62/_remote_module_non_scriptable.py
2023-01-11T20:49:23.2928024Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T20:49:23.2928254Z 
2023-01-11T20:49:23.2928338Z Running tests...
2023-01-11T20:49:23.2928640Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2929042Z   test_shared_allreduce_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2023-01-11T20:49:23.2929249Z 
2023-01-11T20:49:23.2929445Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2929682Z Ran 1 test in 0.001s
2023-01-11T20:49:23.2929886Z 
2023-01-11T20:49:23.2929959Z OK (skipped=1)
2023-01-11T20:49:23.2930066Z 
2023-01-11T20:49:23.2930138Z Generating XML reports...
2023-01-11T20:49:23.2930660Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20230111204903.xml
2023-01-11T20:49:23.2931503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:49:23.2931884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:49:23.2932303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:49:23.2932695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:49:23.2933042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbhqkr9eo
2023-01-11T20:49:23.2933562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbhqkr9eo/_remote_module_non_scriptable.py
2023-01-11T20:49:23.2934029Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T20:49:23.2934262Z 
2023-01-11T20:49:23.2934335Z Running tests...
2023-01-11T20:49:23.2934639Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2935025Z   test_shared_broadcast_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2023-01-11T20:49:23.2935240Z 
2023-01-11T20:49:23.2935436Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2935723Z Ran 1 test in 0.001s
2023-01-11T20:49:23.2935835Z 
2023-01-11T20:49:23.2935906Z OK (skipped=1)
2023-01-11T20:49:23.2935999Z 
2023-01-11T20:49:23.2936083Z Generating XML reports...
2023-01-11T20:49:23.2936690Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20230111204905.xml
2023-01-11T20:49:23.2937294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:49:23.2937612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:49:23.2938090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:49:23.2938437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:49:23.2938822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkmirejtw
2023-01-11T20:49:23.2939209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkmirejtw/_remote_module_non_scriptable.py
2023-01-11T20:49:23.2939714Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T20:49:23.2939908Z 
2023-01-11T20:49:23.2939981Z Running tests...
2023-01-11T20:49:23.2940318Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2940680Z   test_shared_reduce_nccl (__main__.ProcessGroupShareTensorTest) ... skip: At least 2 CUDA GPUS needed (0.000s)
2023-01-11T20:49:23.2940893Z 
2023-01-11T20:49:23.2941120Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2941368Z Ran 1 test in 0.001s
2023-01-11T20:49:23.2941479Z 
2023-01-11T20:49:23.2941537Z OK (skipped=1)
2023-01-11T20:49:23.2941642Z 
2023-01-11T20:49:23.2941727Z Generating XML reports...
2023-01-11T20:49:23.2942200Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20230111204907.xml
2023-01-11T20:49:23.2942794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:49:23.2943110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:49:23.2943537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:49:23.2943952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:49:23.2944281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptybg2joo
2023-01-11T20:49:23.2944734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptybg2joo/_remote_module_non_scriptable.py
2023-01-11T20:49:23.2945194Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T20:49:23.2945388Z 
2023-01-11T20:49:23.2945461Z Running tests...
2023-01-11T20:49:23.2945752Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2946118Z   test_all_gather (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:49:23.2946344Z 
2023-01-11T20:49:23.2946538Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2946756Z Ran 1 test in 0.000s
2023-01-11T20:49:23.2946867Z 
2023-01-11T20:49:23.2946983Z OK (skipped=1)
2023-01-11T20:49:23.2947092Z 
2023-01-11T20:49:23.2947175Z Generating XML reports...
2023-01-11T20:49:23.2947658Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111204908.xml
2023-01-11T20:49:23.2948199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:49:23.2948581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:49:23.2949012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:49:23.2949358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:49:23.2949690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9yb15mw0
2023-01-11T20:49:23.2950088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9yb15mw0/_remote_module_non_scriptable.py
2023-01-11T20:49:23.2950545Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T20:49:23.2950741Z 
2023-01-11T20:49:23.2950801Z Running tests...
2023-01-11T20:49:23.2951103Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2951475Z   test_all_gather_base (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:23.2951703Z 
2023-01-11T20:49:23.2951896Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2952117Z Ran 1 test in 0.001s
2023-01-11T20:49:23.2952229Z 
2023-01-11T20:49:23.2952300Z OK (skipped=1)
2023-01-11T20:49:23.2952407Z 
2023-01-11T20:49:23.2952491Z Generating XML reports...
2023-01-11T20:49:23.2952960Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111204910.xml
2023-01-11T20:49:23.2953512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:49:23.2953843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:49:23.2954268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:49:23.2954599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:49:23.2954941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd_cpn5mv
2023-01-11T20:49:23.2955364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd_cpn5mv/_remote_module_non_scriptable.py
2023-01-11T20:49:23.2955839Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T20:49:23.2956019Z 
2023-01-11T20:49:23.2956091Z Running tests...
2023-01-11T20:49:23.2956388Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2956755Z   test_all_to_all (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:49:23.2957022Z 
2023-01-11T20:49:23.2957203Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2957436Z Ran 1 test in 0.000s
2023-01-11T20:49:23.2957549Z 
2023-01-11T20:49:23.2957620Z OK (skipped=1)
2023-01-11T20:49:23.2957726Z 
2023-01-11T20:49:23.2957810Z Generating XML reports...
2023-01-11T20:49:23.2958280Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111204912.xml
2023-01-11T20:49:23.2958838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:49:23.2959168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:49:23.2959580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:49:23.2959967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:49:23.2960315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkpx7b6h9
2023-01-11T20:49:23.2960722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkpx7b6h9/_remote_module_non_scriptable.py
2023-01-11T20:49:23.2961156Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T20:49:23.2961348Z 
2023-01-11T20:49:23.2961420Z Running tests...
2023-01-11T20:49:23.2961721Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2962080Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:49:23.2962310Z 
2023-01-11T20:49:23.2962501Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2962736Z Ran 1 test in 0.000s
2023-01-11T20:49:23.2962847Z 
2023-01-11T20:49:23.2962918Z OK (skipped=1)
2023-01-11T20:49:23.2963029Z 
2023-01-11T20:49:23.2963099Z Generating XML reports...
2023-01-11T20:49:23.2963627Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111204914.xml
2023-01-11T20:49:23.2964185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:49:23.2964513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:49:23.2964940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:49:23.2965270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:49:23.2965612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2tq92sbs
2023-01-11T20:49:23.2966012Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2tq92sbs/_remote_module_non_scriptable.py
2023-01-11T20:49:23.2966450Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T20:49:23.2966644Z 
2023-01-11T20:49:23.2966717Z Running tests...
2023-01-11T20:49:23.2967017Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2967385Z   test_allreduce (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:49:23.2967610Z 
2023-01-11T20:49:23.2967789Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2968020Z Ran 1 test in 0.000s
2023-01-11T20:49:23.2968131Z 
2023-01-11T20:49:23.2968201Z OK (skipped=1)
2023-01-11T20:49:23.2968307Z 
2023-01-11T20:49:23.2968377Z Generating XML reports...
2023-01-11T20:49:23.2968911Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111204915.xml
2023-01-11T20:49:23.2969473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:49:23.2969845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:49:23.2970257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:49:23.2970598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:49:23.2971080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph_bjl8io
2023-01-11T20:49:23.2971483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph_bjl8io/_remote_module_non_scriptable.py
2023-01-11T20:49:23.2971921Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T20:49:23.2972114Z 
2023-01-11T20:49:23.2972187Z Running tests...
2023-01-11T20:49:23.2972484Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2972900Z   test_broadcast (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:49:23.2973124Z 
2023-01-11T20:49:23.2973320Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2973553Z Ran 1 test in 0.000s
2023-01-11T20:49:23.2973665Z 
2023-01-11T20:49:23.2973740Z OK (skipped=1)
2023-01-11T20:49:23.2973833Z 
2023-01-11T20:49:23.2973917Z Generating XML reports...
2023-01-11T20:49:23.2974400Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111204917.xml
2023-01-11T20:49:23.2974959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:49:23.2975277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:49:23.2975703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:49:23.2976047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:49:23.2976462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_fpwvibg
2023-01-11T20:49:23.2976849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_fpwvibg/_remote_module_non_scriptable.py
2023-01-11T20:49:23.2977306Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T20:49:23.2977500Z 
2023-01-11T20:49:23.2977573Z Running tests...
2023-01-11T20:49:23.2977872Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2978224Z   test_reduce (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:49:23.2978442Z 
2023-01-11T20:49:23.2978638Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2978876Z Ran 1 test in 0.000s
2023-01-11T20:49:23.2978986Z 
2023-01-11T20:49:23.2979043Z OK (skipped=1)
2023-01-11T20:49:23.2979151Z 
2023-01-11T20:49:23.2979238Z Generating XML reports...
2023-01-11T20:49:23.2979716Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111204919.xml
2023-01-11T20:49:23.2980267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:49:23.2980603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:49:23.2981013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:49:23.2981358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:49:23.2981696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ebdia8k
2023-01-11T20:49:23.2982093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ebdia8k/_remote_module_non_scriptable.py
2023-01-11T20:49:23.2982529Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T20:49:23.2982781Z 
2023-01-11T20:49:23.2982853Z Running tests...
2023-01-11T20:49:23.2983152Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2983510Z   test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:23.2983740Z 
2023-01-11T20:49:23.2983936Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2984169Z Ran 1 test in 0.001s
2023-01-11T20:49:23.2984280Z 
2023-01-11T20:49:23.2984350Z OK (skipped=1)
2023-01-11T20:49:23.2984445Z 
2023-01-11T20:49:23.2984529Z Generating XML reports...
2023-01-11T20:49:23.2985007Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111204920.xml
2023-01-11T20:49:23.2985593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:49:23.2985907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:49:23.2986329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:49:23.2986670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:49:23.2987009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyukcm_81
2023-01-11T20:49:23.2987387Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyukcm_81/_remote_module_non_scriptable.py
2023-01-11T20:49:23.2987832Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T20:49:23.2988025Z 
2023-01-11T20:49:23.2988095Z Running tests...
2023-01-11T20:49:23.2988380Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2988768Z   test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:23.2989012Z 
2023-01-11T20:49:23.2989204Z ----------------------------------------------------------------------
2023-01-11T20:49:23.2989436Z Ran 1 test in 0.001s
2023-01-11T20:49:23.2989547Z 
2023-01-11T20:49:23.2989605Z OK (skipped=1)
2023-01-11T20:49:23.2989709Z 
2023-01-11T20:49:23.2989793Z Generating XML reports...
2023-01-11T20:49:23.2990270Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111204922.xml
2023-01-11T20:49:23.2990553Z 
2023-01-11T20:49:23.2990846Z ##[endgroup]
2023-01-11T20:49:23.2991286Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_nccl_n_tuzf3w)
2023-01-11T20:49:23.2991534Z 
2023-01-11T20:49:23.2991753Z Running distributed/_shard/sharded_tensor/ops/test_init ... [2023-01-11 20:49:23.291062]
2023-01-11T20:49:23.2992299Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_init.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:23.291511]
2023-01-11T20:49:24.7192822Z 
2023-01-11T20:49:24.7193373Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_init
2023-01-11T20:49:24.7194579Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_init (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_init_qm6esgvu)
2023-01-11T20:49:24.7195707Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init
2023-01-11T20:49:24.7196113Z 
2023-01-11T20:49:24.7196239Z Running tests...
2023-01-11T20:49:24.7196761Z ----------------------------------------------------------------------
2023-01-11T20:49:24.7197208Z   test_init_sharded_tensor_with_kaiming_uniform (__main__.TestShardedTensorNNInit)
2023-01-11T20:49:24.7197862Z Test torch.nn.init.kaiming_uniform_(ShardedTensor, a, mode, nonlinearit) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:24.7198733Z   test_init_sharded_tensor_with_normal (__main__.TestShardedTensorNNInit)
2023-01-11T20:49:24.7199269Z Test torch.nn.init.normal_(ShardedTensor, mean, std) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:24.7199836Z   test_init_sharded_tensor_with_uniform (__main__.TestShardedTensorNNInit)
2023-01-11T20:49:24.7200424Z Test torch.nn.init.uniform_(ShardedTensor, a, b) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:24.7200742Z 
2023-01-11T20:49:24.7201080Z ----------------------------------------------------------------------
2023-01-11T20:49:24.7201435Z Ran 3 tests in 0.003s
2023-01-11T20:49:24.7201612Z 
2023-01-11T20:49:24.7201720Z OK (skipped=3)
2023-01-11T20:49:24.7201882Z 
2023-01-11T20:49:24.7202019Z Generating XML reports...
2023-01-11T20:49:24.7202993Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20230111204924.xml
2023-01-11T20:49:24.7203416Z 
2023-01-11T20:49:24.7203760Z ##[endgroup]
2023-01-11T20:49:24.7204447Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_init (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_init_qm6esgvu)
2023-01-11T20:49:24.7204850Z 
2023-01-11T20:49:24.7205235Z Running distributed/_shard/sharded_tensor/ops/test_elementwise_ops ... [2023-01-11 20:49:24.719683]
2023-01-11T20:49:24.7206142Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_elementwise_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:24.720110]
2023-01-11T20:49:26.1227221Z 
2023-01-11T20:49:26.1227759Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_elementwise_ops
2023-01-11T20:49:26.1228756Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_elementwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_elementwise_ops_mkaujwvp)
2023-01-11T20:49:26.1230276Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops
2023-01-11T20:49:26.1230565Z 
2023-01-11T20:49:26.1230631Z Running tests...
2023-01-11T20:49:26.1230947Z ----------------------------------------------------------------------
2023-01-11T20:49:26.1231336Z   test_sharded_dropout (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:26.1231914Z   test_sharded_gelu (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:26.1232441Z   test_sharded_relu (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:26.1232975Z   test_sharded_tensor_nan_to_num (__main__.TestShardedTensorElementWiseOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:26.1234716Z 
2023-01-11T20:49:26.1235263Z ----------------------------------------------------------------------
2023-01-11T20:49:26.1235736Z Ran 4 tests in 0.003s
2023-01-11T20:49:26.1235990Z 
2023-01-11T20:49:26.1236148Z OK (skipped=4)
2023-01-11T20:49:26.1236380Z 
2023-01-11T20:49:26.1236567Z Generating XML reports...
2023-01-11T20:49:26.1237788Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20230111204925.xml
2023-01-11T20:49:26.1238492Z 
2023-01-11T20:49:26.1239114Z ##[endgroup]
2023-01-11T20:49:26.1240313Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_elementwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_elementwise_ops_mkaujwvp)
2023-01-11T20:49:26.1240992Z 
2023-01-11T20:49:26.1241440Z Running distributed/_shard/test_partial_tensor ... [2023-01-11 20:49:26.123094]
2023-01-11T20:49:26.1242934Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_partial_tensor.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:26.123539]
2023-01-11T20:49:27.5334910Z 
2023-01-11T20:49:27.5335509Z Expand the folded group to see the log file of distributed/_shard/test_partial_tensor
2023-01-11T20:49:27.5336754Z ##[group]PRINTING LOG FILE of distributed/_shard/test_partial_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_partial_tensor_5m4ghrmv)
2023-01-11T20:49:27.5337821Z Test results will be stored in test-reports/python-unittest/distributed._shard.test_partial_tensor
2023-01-11T20:49:27.5338176Z 
2023-01-11T20:49:27.5338315Z Running tests...
2023-01-11T20:49:27.5338836Z ----------------------------------------------------------------------
2023-01-11T20:49:27.5339168Z   test_cat (__main__.TestPartialTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:27.5339827Z   test_cat_errors (__main__.TestPartialTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:27.5340227Z   test_transpose (__main__.TestPartialTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:49:27.5340624Z   test_partial_tensor_reshard (__main__.TestPartialTensorReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:27.5341038Z   test_partial_tensor_reshard_errors (__main__.TestPartialTensorReshard) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:27.5341267Z 
2023-01-11T20:49:27.5341469Z ----------------------------------------------------------------------
2023-01-11T20:49:27.5341713Z Ran 5 tests in 0.004s
2023-01-11T20:49:27.5341826Z 
2023-01-11T20:49:27.5341884Z OK (skipped=5)
2023-01-11T20:49:27.5341989Z 
2023-01-11T20:49:27.5342072Z Generating XML reports...
2023-01-11T20:49:27.5342550Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20230111204927.xml
2023-01-11T20:49:27.5343285Z Generated XML report: test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20230111204927.xml
2023-01-11T20:49:27.5343657Z 
2023-01-11T20:49:27.5343896Z ##[endgroup]
2023-01-11T20:49:27.5344365Z FINISHED PRINTING LOG FILE of distributed/_shard/test_partial_tensor (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_partial_tensor_5m4ghrmv)
2023-01-11T20:49:27.5344625Z 
2023-01-11T20:49:27.5344852Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops ... [2023-01-11 20:49:27.533872]
2023-01-11T20:49:27.5345415Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:27.534215]
2023-01-11T20:49:28.9478925Z 
2023-01-11T20:49:28.9479522Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_tensor_ops
2023-01-11T20:49:28.9480798Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_tensor_ops_r9l65f7g)
2023-01-11T20:49:28.9481988Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops
2023-01-11T20:49:28.9482281Z 
2023-01-11T20:49:28.9482354Z Running tests...
2023-01-11T20:49:28.9482661Z ----------------------------------------------------------------------
2023-01-11T20:49:28.9482990Z   test_clone (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:28.9483451Z   test_deep_copy (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:28.9483896Z   test_detach (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:28.9484403Z   test_inplace_copy (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:28.9484969Z   test_set_requires_grad (__main__.TestTensorOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:28.9485166Z 
2023-01-11T20:49:28.9485372Z ----------------------------------------------------------------------
2023-01-11T20:49:28.9485611Z Ran 5 tests in 0.003s
2023-01-11T20:49:28.9485723Z 
2023-01-11T20:49:28.9485793Z OK (skipped=5)
2023-01-11T20:49:28.9485899Z 
2023-01-11T20:49:28.9485970Z Generating XML reports...
2023-01-11T20:49:28.9486447Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20230111204928.xml
2023-01-11T20:49:28.9486711Z 
2023-01-11T20:49:28.9486952Z ##[endgroup]
2023-01-11T20:49:28.9487532Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_tensor_ops_r9l65f7g)
2023-01-11T20:49:28.9487829Z 
2023-01-11T20:49:28.9488134Z Running distributed/_shard/sharded_tensor/ops/test_linear ... [2023-01-11 20:49:28.948251]
2023-01-11T20:49:28.9488694Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_linear.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:28.948632]
2023-01-11T20:49:30.3572979Z 
2023-01-11T20:49:30.3573502Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_linear
2023-01-11T20:49:30.3574682Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_linear (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_linear_ujljzk4b)
2023-01-11T20:49:30.3575380Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear
2023-01-11T20:49:30.3575746Z 
2023-01-11T20:49:30.3575884Z Running tests...
2023-01-11T20:49:30.3576439Z ----------------------------------------------------------------------
2023-01-11T20:49:30.3576862Z   test_sharded_linear_colwise (__main__.TestShardedTensorOpsLinear) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:30.3577402Z   test_sharded_linear_errors (__main__.TestShardedTensorOpsLinear) ... skip: c10d was not compiled with the NCCL backend (0.003s)
2023-01-11T20:49:30.3577876Z   test_sharded_linear_rowwise (__main__.TestShardedTensorOpsLinear) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:30.3578136Z 
2023-01-11T20:49:30.3578349Z ----------------------------------------------------------------------
2023-01-11T20:49:30.3578576Z Ran 3 tests in 0.005s
2023-01-11T20:49:30.3578689Z 
2023-01-11T20:49:30.3578763Z OK (skipped=3)
2023-01-11T20:49:30.3578870Z 
2023-01-11T20:49:30.3578953Z Generating XML reports...
2023-01-11T20:49:30.3579451Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20230111204930.xml
2023-01-11T20:49:30.3579750Z 
2023-01-11T20:49:30.3579986Z ##[endgroup]
2023-01-11T20:49:30.3580526Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_linear (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_linear_ujljzk4b)
2023-01-11T20:49:30.3580838Z 
2023-01-11T20:49:30.3581090Z Running distributed/_shard/sharding_plan/test_sharding_plan ... [2023-01-11 20:49:30.357630]
2023-01-11T20:49:30.3581819Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:30.357988]
2023-01-11T20:49:31.7731002Z 
2023-01-11T20:49:31.7731625Z Expand the folded group to see the log file of distributed/_shard/sharding_plan/test_sharding_plan
2023-01-11T20:49:31.7732816Z ##[group]PRINTING LOG FILE of distributed/_shard/sharding_plan/test_sharding_plan (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_plan-test_sharding_plan_wba8y1qb)
2023-01-11T20:49:31.7733769Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan
2023-01-11T20:49:31.7733990Z 
2023-01-11T20:49:31.7734065Z Running tests...
2023-01-11T20:49:31.7734370Z ----------------------------------------------------------------------
2023-01-11T20:49:31.7734733Z   test_custom_sharding_planner (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:31.7735211Z   test_reshard_to_ddp_sharding_plan (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:31.7735611Z   test_shard_module_sub_process_group (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:31.7736005Z   test_sharding_plan_errors (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:31.7736551Z   test_sharding_plan_simple_megatron (__main__.TestShardingPlan) ... skip: c10d was not compiled with the NCCL backend (0.003s)
2023-01-11T20:49:31.7736771Z 
2023-01-11T20:49:31.7736974Z ----------------------------------------------------------------------
2023-01-11T20:49:31.7737213Z Ran 5 tests in 0.007s
2023-01-11T20:49:31.7737325Z 
2023-01-11T20:49:31.7737395Z OK (skipped=5)
2023-01-11T20:49:31.7737501Z 
2023-01-11T20:49:31.7737571Z Generating XML reports...
2023-01-11T20:49:31.7738174Z Generated XML report: test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20230111204931.xml
2023-01-11T20:49:31.7738484Z 
2023-01-11T20:49:31.7738731Z ##[endgroup]
2023-01-11T20:49:31.7739263Z FINISHED PRINTING LOG FILE of distributed/_shard/sharding_plan/test_sharding_plan (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharding_plan-test_sharding_plan_wba8y1qb)
2023-01-11T20:49:31.7739553Z 
2023-01-11T20:49:31.7739791Z Running distributed/_shard/sharded_tensor/ops/test_matrix_ops ... [2023-01-11 20:49:31.773384]
2023-01-11T20:49:31.7740354Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_matrix_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:31.773732]
2023-01-11T20:49:33.1929020Z 
2023-01-11T20:49:33.1929623Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_matrix_ops
2023-01-11T20:49:33.1931060Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_matrix_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_matrix_ops_uxm2z461)
2023-01-11T20:49:33.1932171Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops
2023-01-11T20:49:33.1932533Z 
2023-01-11T20:49:33.1932609Z Running tests...
2023-01-11T20:49:33.1932916Z ----------------------------------------------------------------------
2023-01-11T20:49:33.1933385Z   test_sharded_tensor_contiguous (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:33.1933976Z   test_sharded_tensor_layer_norm (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:33.1934491Z   test_sharded_tensor_layer_norm_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:33.1934961Z   test_sharded_tensor_masked_fill (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:49:33.1935387Z   test_sharded_tensor_masked_fill_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:33.1935860Z   test_sharded_tensor_softmax (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:33.1936338Z   test_sharded_tensor_transpose (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:33.1937046Z   test_sharded_tensor_transpose_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:49:33.1937572Z   test_sharded_tensor_type_as (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:33.1937990Z   test_sharded_tensor_view (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:33.1938406Z   test_sharded_tensor_view_error (__main__.TestShardedTensorMatrixOps) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:33.1938628Z 
2023-01-11T20:49:33.1938842Z ----------------------------------------------------------------------
2023-01-11T20:49:33.1939068Z Ran 11 tests in 0.008s
2023-01-11T20:49:33.1939181Z 
2023-01-11T20:49:33.1939251Z OK (skipped=11)
2023-01-11T20:49:33.1939357Z 
2023-01-11T20:49:33.1939439Z Generating XML reports...
2023-01-11T20:49:33.1940042Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops/TEST-TestShardedTensorMatrixOps-20230111204932.xml
2023-01-11T20:49:33.1940341Z 
2023-01-11T20:49:33.1940601Z ##[endgroup]
2023-01-11T20:49:33.1941124Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_matrix_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_matrix_ops_uxm2z461)
2023-01-11T20:49:33.1941415Z 
2023-01-11T20:49:33.1941625Z Running distributed/_composable/test_checkpoint ... [2023-01-11 20:49:33.193278]
2023-01-11T20:49:33.1942143Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:33.193634]
2023-01-11T20:49:34.9619240Z 
2023-01-11T20:49:34.9619802Z Expand the folded group to see the log file of distributed/_composable/test_checkpoint
2023-01-11T20:49:34.9620959Z ##[group]PRINTING LOG FILE of distributed/_composable/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_checkpoint_uj3it30n)
2023-01-11T20:49:34.9621920Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_checkpoint
2023-01-11T20:49:34.9622131Z 
2023-01-11T20:49:34.9622206Z Running tests...
2023-01-11T20:49:34.9622559Z ----------------------------------------------------------------------
2023-01-11T20:49:34.9622856Z   test_random_cpu (__main__.TestCheckpoint) ... ok (0.013s)
2023-01-11T20:49:34.9623191Z   test_tensor_only_cpu_use_reentrant_False (__main__.TestCheckpoint) ... ok (0.004s)
2023-01-11T20:49:34.9623548Z   test_tensor_only_cpu_use_reentrant_True (__main__.TestCheckpoint) ... ok (0.004s)
2023-01-11T20:49:34.9624008Z   test_tensor_only_gpu_use_reentrant_False (__main__.TestCheckpoint) ... skip: no cuda (0.000s)
2023-01-11T20:49:34.9624457Z   test_tensor_only_gpu_use_reentrant_True (__main__.TestCheckpoint) ... skip: no cuda (0.000s)
2023-01-11T20:49:34.9624661Z 
2023-01-11T20:49:34.9624910Z ----------------------------------------------------------------------
2023-01-11T20:49:34.9625152Z Ran 5 tests in 0.022s
2023-01-11T20:49:34.9625253Z 
2023-01-11T20:49:34.9625325Z OK (skipped=2)
2023-01-11T20:49:34.9625465Z 
2023-01-11T20:49:34.9625568Z Generating XML reports...
2023-01-11T20:49:34.9626034Z Generated XML report: test-reports/python-unittest/distributed._composable.test_checkpoint/TEST-TestCheckpoint-20230111204934.xml
2023-01-11T20:49:34.9626345Z 
2023-01-11T20:49:34.9626573Z ##[endgroup]
2023-01-11T20:49:34.9627103Z FINISHED PRINTING LOG FILE of distributed/_composable/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_checkpoint_uj3it30n)
2023-01-11T20:49:34.9627471Z 
2023-01-11T20:49:34.9627700Z Running distributed/test_data_parallel ... [2023-01-11 20:49:34.962301]
2023-01-11T20:49:34.9628554Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_data_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:34.962682]
2023-01-11T20:49:36.7208672Z 
2023-01-11T20:49:36.7209248Z Expand the folded group to see the log file of distributed/test_data_parallel
2023-01-11T20:49:36.7210756Z ##[group]PRINTING LOG FILE of distributed/test_data_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-test_data_parallel_5qlw96zx)
2023-01-11T20:49:36.7211537Z Test results will be stored in test-reports/python-unittest/distributed.test_data_parallel
2023-01-11T20:49:36.7211736Z 
2023-01-11T20:49:36.7211811Z Running tests...
2023-01-11T20:49:36.7212120Z ----------------------------------------------------------------------
2023-01-11T20:49:36.7212508Z   test_autocast (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2023-01-11T20:49:36.7212966Z   test_data_parallel (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2023-01-11T20:49:36.7213586Z   test_data_parallel_buffers_requiring_grad (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2023-01-11T20:49:36.7214254Z   test_data_parallel_complex (__main__.TestDataParallel) ... skip: At least 2 CUDA GPUS needed (0.001s)
2023-01-11T20:49:36.7214745Z   test_data_parallel_device_args (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2023-01-11T20:49:36.7215378Z   test_data_parallel_function_deletion (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2023-01-11T20:49:36.7216167Z   test_data_parallel_lazy_linear (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2023-01-11T20:49:36.7216822Z   test_data_parallel_model_device (__main__.TestDataParallel)
2023-01-11T20:49:36.7217474Z Test device[0] check at forward time. ... skip: multi-GPU not supported (0.003s)
2023-01-11T20:49:36.7218206Z   test_data_parallel_model_no_refcycles (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2023-01-11T20:49:36.7219032Z   test_data_parallel_module_zero_inputs (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2023-01-11T20:49:36.7219485Z   test_data_parallel_multiple_input (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.002s)
2023-01-11T20:49:36.7219930Z   test_data_parallel_nested_input (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2023-01-11T20:49:36.7220373Z   test_data_parallel_nested_output (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2023-01-11T20:49:36.7220807Z   test_data_parallel_no_grad (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2023-01-11T20:49:36.7221218Z   test_data_parallel_rnn (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2023-01-11T20:49:36.7221644Z   test_data_parallel_small_back (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2023-01-11T20:49:36.7222068Z   test_data_parallel_sparse (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2023-01-11T20:49:36.7222476Z   test_gather_cpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2023-01-11T20:49:36.7222893Z   test_gather_different_len_dicts (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2023-01-11T20:49:36.7223300Z   test_gather_gpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2023-01-11T20:49:36.7223704Z   test_parallel_apply (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2023-01-11T20:49:36.7224114Z   test_parallel_apply_autocast (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2023-01-11T20:49:36.7224478Z   test_parallel_apply_passes_exception (__main__.TestDataParallel) ... skip: CUDA unavailable (0.000s)
2023-01-11T20:49:36.7224923Z   test_parameter_list_dict_replica (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2023-01-11T20:49:36.7225332Z   test_replicate (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2023-01-11T20:49:36.7225726Z   test_replicate_buffers (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2023-01-11T20:49:36.7226274Z   test_save_replica_module (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2023-01-11T20:49:36.7226680Z   test_scatter_cpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2023-01-11T20:49:36.7227085Z   test_scatter_gpu (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.000s)
2023-01-11T20:49:36.7227482Z   test_strided_grad_layout (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.003s)
2023-01-11T20:49:36.7227884Z   test_zero_grad (__main__.TestDataParallel) ... skip: multi-GPU not supported (0.001s)
2023-01-11T20:49:36.7228259Z   test_data_parallel_module_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2023-01-11T20:49:36.7228650Z   test_data_parallel_module_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2023-01-11T20:49:36.7229087Z   test_data_parallel_module_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2023-01-11T20:49:36.7229507Z   test_data_parallel_module_kwargs_only_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2023-01-11T20:49:36.7229962Z   test_data_parallel_module_kwargs_only_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2023-01-11T20:49:36.7230382Z   test_data_parallel_module_kwargs_only_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2023-01-11T20:49:36.7230804Z   test_data_parallel_module_kwargs_only_empty_dict_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2023-01-11T20:49:36.7231250Z   test_data_parallel_module_kwargs_only_empty_dict_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2023-01-11T20:49:36.7231690Z   test_data_parallel_module_kwargs_only_empty_dict_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2023-01-11T20:49:36.7232134Z   test_data_parallel_module_kwargs_only_empty_list_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2023-01-11T20:49:36.7232553Z   test_data_parallel_module_kwargs_only_empty_list_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2023-01-11T20:49:36.7232991Z   test_data_parallel_module_kwargs_only_empty_list_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2023-01-11T20:49:36.7233436Z   test_data_parallel_module_kwargs_only_empty_tuple_cpu_float16 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2023-01-11T20:49:36.7233881Z   test_data_parallel_module_kwargs_only_empty_tuple_cpu_float32 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2023-01-11T20:49:36.7234312Z   test_data_parallel_module_kwargs_only_empty_tuple_cpu_float64 (__main__.TestDataParallelDeviceTypeCPU) ... skip: Only runs on cuda (0.001s)
2023-01-11T20:49:36.7234552Z 
2023-01-11T20:49:36.7234756Z ----------------------------------------------------------------------
2023-01-11T20:49:36.7234997Z Ran 46 tests in 0.044s
2023-01-11T20:49:36.7235109Z 
2023-01-11T20:49:36.7235181Z OK (skipped=46)
2023-01-11T20:49:36.7235275Z 
2023-01-11T20:49:36.7235359Z Generating XML reports...
2023-01-11T20:49:36.7235802Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20230111204936.xml
2023-01-11T20:49:36.7236408Z Generated XML report: test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCPU-20230111204936.xml
2023-01-11T20:49:36.7236691Z 
2023-01-11T20:49:36.7236961Z ##[endgroup]
2023-01-11T20:49:36.7237399Z FINISHED PRINTING LOG FILE of distributed/test_data_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-test_data_parallel_5qlw96zx)
2023-01-11T20:49:36.7237650Z 
2023-01-11T20:49:36.7237851Z Running distributed/nn/jit/test_instantiator ... [2023-01-11 20:49:36.721244]
2023-01-11T20:49:36.7238414Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:36.721650]
2023-01-11T20:49:38.4512779Z 
2023-01-11T20:49:38.4513412Z Expand the folded group to see the log file of distributed/nn/jit/test_instantiator
2023-01-11T20:49:38.4514686Z ##[group]PRINTING LOG FILE of distributed/nn/jit/test_instantiator (/var/lib/jenkins/workspace/test/test-reports/distributed-nn-jit-test_instantiator_fkt0yze9)
2023-01-11T20:49:38.4515617Z Test results will be stored in test-reports/python-unittest/distributed.nn.jit.test_instantiator
2023-01-11T20:49:38.4515829Z 
2023-01-11T20:49:38.4515907Z Running tests...
2023-01-11T20:49:38.4516219Z ----------------------------------------------------------------------
2023-01-11T20:49:38.4516544Z   test_get_arg_return_types_from_interface (__main__.TestInstantiator) ... ok (0.250s)
2023-01-11T20:49:38.4517075Z   test_instantiate_non_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.002s)
2023-01-11T20:49:38.4517586Z   test_instantiate_scripted_remote_module_template (__main__.TestInstantiator) ... ok (0.010s)
2023-01-11T20:49:38.4517788Z 
2023-01-11T20:49:38.4517995Z ----------------------------------------------------------------------
2023-01-11T20:49:38.4518226Z Ran 3 tests in 0.262s
2023-01-11T20:49:38.4518343Z 
2023-01-11T20:49:38.4518406Z OK
2023-01-11T20:49:38.4518498Z 
2023-01-11T20:49:38.4518583Z Generating XML reports...
2023-01-11T20:49:38.4519050Z Generated XML report: test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20230111204937.xml
2023-01-11T20:49:38.4519296Z 
2023-01-11T20:49:38.4519547Z ##[endgroup]
2023-01-11T20:49:38.4520016Z FINISHED PRINTING LOG FILE of distributed/nn/jit/test_instantiator (/var/lib/jenkins/workspace/test/test-reports/distributed-nn-jit-test_instantiator_fkt0yze9)
2023-01-11T20:49:38.4520282Z 
2023-01-11T20:49:38.4520475Z Running distributed/test_launcher ... [2023-01-11 20:49:38.451655]
2023-01-11T20:49:38.4521679Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_launcher.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:38.452002]
2023-01-11T20:49:40.5463876Z 
2023-01-11T20:49:40.5464397Z Expand the folded group to see the log file of distributed/test_launcher
2023-01-11T20:49:40.5465150Z ##[group]PRINTING LOG FILE of distributed/test_launcher (/var/lib/jenkins/workspace/test/test-reports/distributed-test_launcher_8rn3cmil)
2023-01-11T20:49:40.5465768Z Test results will be stored in test-reports/python-unittest/distributed.test_launcher
2023-01-11T20:49:40.5465946Z 
2023-01-11T20:49:40.5466020Z Running tests...
2023-01-11T20:49:40.5466326Z ----------------------------------------------------------------------
2023-01-11T20:49:40.5467411Z   test_launch_user_script (__main__.TestDistributedLaunch) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79488 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.247s)
2023-01-11T20:49:40.5468028Z 
2023-01-11T20:49:40.5468279Z ----------------------------------------------------------------------
2023-01-11T20:49:40.5468571Z Ran 1 test in 0.248s
2023-01-11T20:49:40.5468733Z 
2023-01-11T20:49:40.5468810Z OK (skipped=1)
2023-01-11T20:49:40.5468914Z 
2023-01-11T20:49:40.5468999Z Generating XML reports...
2023-01-11T20:49:40.5469484Z Generated XML report: test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20230111204940.xml
2023-01-11T20:49:40.5469748Z 
2023-01-11T20:49:40.5469966Z ##[endgroup]
2023-01-11T20:49:40.5470432Z FINISHED PRINTING LOG FILE of distributed/test_launcher (/var/lib/jenkins/workspace/test/test-reports/distributed-test_launcher_8rn3cmil)
2023-01-11T20:49:40.5470670Z 
2023-01-11T20:49:40.5470951Z Running distributed/elastic/metrics/api_test ... [2023-01-11 20:49:40.546695]
2023-01-11T20:49:40.5472026Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/metrics/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:40.547044]
2023-01-11T20:49:42.2509102Z 
2023-01-11T20:49:42.2509990Z Expand the folded group to see the log file of distributed/elastic/metrics/api_test
2023-01-11T20:49:42.2510937Z ##[group]PRINTING LOG FILE of distributed/elastic/metrics/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-metrics-api_test_6pz5np_v)
2023-01-11T20:49:42.2511579Z Test results will be stored in test-reports/python-unittest/distributed.elastic.metrics.api_test
2023-01-11T20:49:42.2511814Z 
2023-01-11T20:49:42.2511888Z Running tests...
2023-01-11T20:49:42.2512201Z ----------------------------------------------------------------------
2023-01-11T20:49:42.2512510Z   test_get_metric_name (__main__.MetricsApiTest) ... ok (0.246s)
2023-01-11T20:49:42.2513029Z   test_inheritance (__main__.MetricsApiTest) ... ok (0.001s)
2023-01-11T20:49:42.2513401Z   test_profile (__main__.MetricsApiTest) ... ok (0.001s)
2023-01-11T20:49:42.2513550Z 
2023-01-11T20:49:42.2513805Z ----------------------------------------------------------------------
2023-01-11T20:49:42.2514045Z Ran 3 tests in 0.249s
2023-01-11T20:49:42.2514158Z 
2023-01-11T20:49:42.2514206Z OK
2023-01-11T20:49:42.2514297Z 
2023-01-11T20:49:42.2514381Z Generating XML reports...
2023-01-11T20:49:42.2514900Z Generated XML report: test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20230111204941.xml
2023-01-11T20:49:42.2515166Z 
2023-01-11T20:49:42.2515439Z ##[endgroup]
2023-01-11T20:49:42.2516083Z FINISHED PRINTING LOG FILE of distributed/elastic/metrics/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-metrics-api_test_6pz5np_v)
2023-01-11T20:49:42.2516441Z 
2023-01-11T20:49:42.2516658Z Running distributed/checkpoint/test_planner ... [2023-01-11 20:49:42.251175]
2023-01-11T20:49:42.2517191Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_planner.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:42.251510]
2023-01-11T20:49:44.0367047Z 
2023-01-11T20:49:44.0367623Z Expand the folded group to see the log file of distributed/checkpoint/test_planner
2023-01-11T20:49:44.0368387Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_planner (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_planner_ffpblxvt)
2023-01-11T20:49:44.0369021Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_planner
2023-01-11T20:49:44.0369240Z 
2023-01-11T20:49:44.0369300Z Running tests...
2023-01-11T20:49:44.0369610Z ----------------------------------------------------------------------
2023-01-11T20:49:44.0369898Z   test_global_plan (__main__.TestSavePlan) ... ok (0.255s)
2023-01-11T20:49:44.0370209Z   test_load_with_resharding (__main__.TestSavePlan) ... ok (0.004s)
2023-01-11T20:49:44.0370506Z   test_load_with_world_size_diff_by_one (__main__.TestSavePlan) ... ok (0.003s)
2023-01-11T20:49:44.0370999Z   test_local_load_plan (__main__.TestSavePlan) ... ok (0.003s)
2023-01-11T20:49:44.0371274Z   test_local_plan (__main__.TestSavePlan) ... ok (0.003s)
2023-01-11T20:49:44.0371420Z 
2023-01-11T20:49:44.0371612Z ----------------------------------------------------------------------
2023-01-11T20:49:44.0371855Z Ran 5 tests in 0.267s
2023-01-11T20:49:44.0371970Z 
2023-01-11T20:49:44.0372034Z OK
2023-01-11T20:49:44.0372126Z 
2023-01-11T20:49:44.0372197Z Generating XML reports...
2023-01-11T20:49:44.0372657Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_planner/TEST-TestSavePlan-20230111204943.xml
2023-01-11T20:49:44.0372915Z 
2023-01-11T20:49:44.0373139Z ##[endgroup]
2023-01-11T20:49:44.0373626Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_planner (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_planner_ffpblxvt)
2023-01-11T20:49:44.0374216Z 
2023-01-11T20:49:44.0374553Z Running distributed/_composable/test_contract ... [2023-01-11 20:49:44.036968]
2023-01-11T20:49:44.0375099Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_contract.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:44.037316]
2023-01-11T20:49:45.8039722Z 
2023-01-11T20:49:45.8040186Z Expand the folded group to see the log file of distributed/_composable/test_contract
2023-01-11T20:49:45.8041169Z ##[group]PRINTING LOG FILE of distributed/_composable/test_contract (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_contract_zfrq580r)
2023-01-11T20:49:45.8041869Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_contract
2023-01-11T20:49:45.8042123Z 
2023-01-11T20:49:45.8042202Z Running tests...
2023-01-11T20:49:45.8042501Z ----------------------------------------------------------------------
2023-01-11T20:49:45.8042987Z   test_add_hooks (__main__.TestContract) ... ok (0.250s)
2023-01-11T20:49:45.8043254Z   test_modify_fqn (__main__.TestContract) ... ok (0.002s)
2023-01-11T20:49:45.8043493Z   test_registry (__main__.TestContract) ... ok (0.002s)
2023-01-11T20:49:45.8043772Z   test_state (__main__.TestContract) ... ok (0.002s)
2023-01-11T20:49:45.8043908Z 
2023-01-11T20:49:45.8044151Z ----------------------------------------------------------------------
2023-01-11T20:49:45.8044420Z Ran 4 tests in 0.256s
2023-01-11T20:49:45.8044570Z 
2023-01-11T20:49:45.8044632Z OK
2023-01-11T20:49:45.8044721Z 
2023-01-11T20:49:45.8044840Z Generating XML reports...
2023-01-11T20:49:45.8045310Z Generated XML report: test-reports/python-unittest/distributed._composable.test_contract/TEST-TestContract-20230111204945.xml
2023-01-11T20:49:45.8045548Z 
2023-01-11T20:49:45.8045782Z ##[endgroup]
2023-01-11T20:49:45.8046240Z FINISHED PRINTING LOG FILE of distributed/_composable/test_contract (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_contract_zfrq580r)
2023-01-11T20:49:45.8046511Z 
2023-01-11T20:49:45.8046724Z Running distributed/elastic/utils/logging_test ... [2023-01-11 20:49:45.804329]
2023-01-11T20:49:45.8048742Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/logging_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:45.804670]
2023-01-11T20:49:47.6131528Z 
2023-01-11T20:49:47.6132097Z Expand the folded group to see the log file of distributed/elastic/utils/logging_test
2023-01-11T20:49:47.6133287Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/logging_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-logging_test_e_b3u6rf)
2023-01-11T20:49:47.6134210Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.logging_test
2023-01-11T20:49:47.6134435Z 
2023-01-11T20:49:47.6134495Z Running tests...
2023-01-11T20:49:47.6134806Z ----------------------------------------------------------------------
2023-01-11T20:49:47.6135115Z   test_derive_module_name (__main__.LoggingTest) ... ok (0.272s)
2023-01-11T20:49:47.6135390Z   test_logger_name (__main__.LoggingTest) ... ok (0.002s)
2023-01-11T20:49:47.6135561Z 
2023-01-11T20:49:47.6135760Z ----------------------------------------------------------------------
2023-01-11T20:49:47.6135998Z Ran 2 tests in 0.274s
2023-01-11T20:49:47.6136108Z 
2023-01-11T20:49:47.6136189Z OK
2023-01-11T20:49:47.6136394Z 
2023-01-11T20:49:47.6136515Z Generating XML reports...
2023-01-11T20:49:47.6137050Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20230111204947.xml
2023-01-11T20:49:47.6137312Z 
2023-01-11T20:49:47.6137552Z ##[endgroup]
2023-01-11T20:49:47.6138010Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/logging_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-logging_test_e_b3u6rf)
2023-01-11T20:49:47.6138292Z 
2023-01-11T20:49:47.6138504Z Running distributed/_tensor/test_pointwise_ops ... [2023-01-11 20:49:47.613514]
2023-01-11T20:49:47.6140305Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_pointwise_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:47.613861]
2023-01-11T20:49:49.4484123Z 
2023-01-11T20:49:49.4484753Z Expand the folded group to see the log file of distributed/_tensor/test_pointwise_ops
2023-01-11T20:49:49.4485869Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_pointwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_pointwise_ops_f41wajng)
2023-01-11T20:49:49.4486982Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_pointwise_ops
2023-01-11T20:49:49.4487358Z 
2023-01-11T20:49:49.4487476Z Running tests...
2023-01-11T20:49:49.4488033Z ----------------------------------------------------------------------
2023-01-11T20:49:49.4488754Z   test_activations (__main__.DistElementwiseOpsTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:49:49.4489488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:49:49.4489845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:49:49.4490356Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4490758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:49:49.4491468Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4491973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4492490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4492781Z ok (0.286s)
2023-01-11T20:49:49.4493101Z   test_dropout (__main__.DistElementwiseOpsTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:49:49.4493521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:49:49.4493880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:49:49.4494263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:49:49.4494736Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4495247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4495755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4496328Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4496811Z INFO:torch.testing._internal.common_distributed:Thread 3 skipping test test_dropout for following reason: testing RNG based ops is broken: https://github.com/pytorch/tau/issues/494
2023-01-11T20:49:49.4497343Z INFO:torch.testing._internal.common_distributed:Thread 0 skipping test test_dropout for following reason: testing RNG based ops is broken: https://github.com/pytorch/tau/issues/494
2023-01-11T20:49:49.4497875Z INFO:torch.testing._internal.common_distributed:Thread 1 skipping test test_dropout for following reason: testing RNG based ops is broken: https://github.com/pytorch/tau/issues/494
2023-01-11T20:49:49.4498401Z INFO:torch.testing._internal.common_distributed:Thread 2 skipping test test_dropout for following reason: testing RNG based ops is broken: https://github.com/pytorch/tau/issues/494
2023-01-11T20:49:49.4498922Z skip: Test skipped at subprocess level, look at subprocess log for skip reason (0.012s)
2023-01-11T20:49:49.4499330Z   test_dropout_backward (__main__.DistElementwiseOpsTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:49:49.4499759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:49:49.4500122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:49:49.4500616Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4500995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:49:49.4501477Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4502037Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4502576Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4502887Z skip: Need at least 4 CUDA devices (0.012s)
2023-01-11T20:49:49.4503253Z   test_dropout_errors (__main__.DistElementwiseOpsTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:49:49.4503673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:49:49.4504019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:49:49.4504500Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4504886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:49:49.4505369Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4505863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4506369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4506646Z ok (0.012s)
2023-01-11T20:49:49.4506967Z   test_mul_out (__main__.DistElementwiseOpsTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:49:49.4507364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:49:49.4507715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:49:49.4508198Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4508571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:49:49.4509051Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4510814Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4511652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:49.4512177Z ok (0.013s)
2023-01-11T20:49:49.4512349Z 
2023-01-11T20:49:49.4512735Z ----------------------------------------------------------------------
2023-01-11T20:49:49.4513184Z Ran 5 tests in 0.335s
2023-01-11T20:49:49.4513394Z 
2023-01-11T20:49:49.4513527Z OK (skipped=2)
2023-01-11T20:49:49.4513831Z 
2023-01-11T20:49:49.4513970Z Generating XML reports...
2023-01-11T20:49:49.4514887Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_pointwise_ops/TEST-DistElementwiseOpsTest-20230111204948.xml
2023-01-11T20:49:49.4515395Z 
2023-01-11T20:49:49.4515857Z ##[endgroup]
2023-01-11T20:49:49.4516695Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_pointwise_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_pointwise_ops_f41wajng)
2023-01-11T20:49:49.4517173Z 
2023-01-11T20:49:49.4517536Z Running distributed/elastic/utils/util_test ... [2023-01-11 20:49:49.448748]
2023-01-11T20:49:49.4572048Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/util_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:49.449117]
2023-01-11T20:49:51.2759783Z 
2023-01-11T20:49:51.2760383Z Expand the folded group to see the log file of distributed/elastic/utils/util_test
2023-01-11T20:49:51.2761939Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/util_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-util_test_q1h9j2fq)
2023-01-11T20:49:51.2762978Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.util_test
2023-01-11T20:49:51.2763296Z 
2023-01-11T20:49:51.2763407Z Running tests...
2023-01-11T20:49:51.2763911Z ----------------------------------------------------------------------
2023-01-11T20:49:51.2764373Z   test_get_all_rank_0 (__main__.StoreUtilTest) ... ok (0.249s)
2023-01-11T20:49:51.2764812Z   test_get_all_rank_n (__main__.StoreUtilTest) ... ok (0.002s)
2023-01-11T20:49:51.2765231Z   test_synchronize (__main__.StoreUtilTest) ... ok (0.002s)
2023-01-11T20:49:51.2765608Z   test_get_logger (__main__.UtilTest) ... ok (0.082s)
2023-01-11T20:49:51.2766019Z   test_get_logger_custom_name (__main__.UtilTest) ... ok (0.001s)
2023-01-11T20:49:51.2766461Z   test_get_logger_different (__main__.UtilTest) ... ok (0.001s)
2023-01-11T20:49:51.2766873Z   test_get_logger_none (__main__.UtilTest) ... ok (0.001s)
2023-01-11T20:49:51.2767091Z 
2023-01-11T20:49:51.2767408Z ----------------------------------------------------------------------
2023-01-11T20:49:51.2767783Z Ran 7 tests in 0.337s
2023-01-11T20:49:51.2767955Z 
2023-01-11T20:49:51.2768046Z OK
2023-01-11T20:49:51.2768194Z 
2023-01-11T20:49:51.2768317Z Generating XML reports...
2023-01-11T20:49:51.2769057Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20230111204950.xml
2023-01-11T20:49:51.2769851Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20230111204950.xml
2023-01-11T20:49:51.2770173Z 
2023-01-11T20:49:51.2770549Z ##[endgroup]
2023-01-11T20:49:51.2771540Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/util_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-util_test_q1h9j2fq)
2023-01-11T20:49:51.2771955Z 
2023-01-11T20:49:51.2772310Z Running distributed/fsdp/test_checkpoint_wrapper ... [2023-01-11 20:49:51.276283]
2023-01-11T20:49:51.2773157Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:51.276716]
2023-01-11T20:49:53.1359742Z 
2023-01-11T20:49:53.1360381Z Expand the folded group to see the log file of distributed/fsdp/test_checkpoint_wrapper
2023-01-11T20:49:53.1361620Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_checkpoint_wrapper_carw9pv0)
2023-01-11T20:49:53.1362772Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper
2023-01-11T20:49:53.1363146Z 
2023-01-11T20:49:53.1363288Z Running tests...
2023-01-11T20:49:53.1363849Z ----------------------------------------------------------------------
2023-01-11T20:49:53.1364294Z   test_apply_activation_checkpointing (__main__.CheckpointWrapperTest)
2023-01-11T20:49:53.1364889Z Ensures that `apply_activation_checkpointing` can be used ... ok (0.333s)
2023-01-11T20:49:53.1365243Z   test_checkpoint_wrapper_cpu_offload (__main__.CheckpointWrapperTest) ... skip: Test requires CUDA (0.001s)
2023-01-11T20:49:53.1365607Z   test_checkpoint_wrapper_kwarg_support (__main__.CheckpointWrapperTest) ... ok (0.007s)
2023-01-11T20:49:53.1365936Z   test_checkpoint_wrapper_parity (__main__.CheckpointWrapperTest)
2023-01-11T20:49:53.1366243Z Tests that using checkpoint_wrapper or the functional ... skip: Test requires CUDA (0.002s)
2023-01-11T20:49:53.1366576Z   test_forward_missing_attributes (__main__.CheckpointWrapperTest) ... ok (0.001s)
2023-01-11T20:49:53.1366873Z   test_fqn (__main__.CheckpointWrapperTest) ... ok (0.001s)
2023-01-11T20:49:53.1367171Z   test_load_activation_checkpointed_module (__main__.CheckpointWrapperTest) ... ok (0.002s)
2023-01-11T20:49:53.1367364Z 
2023-01-11T20:49:53.1367570Z ----------------------------------------------------------------------
2023-01-11T20:49:53.1367886Z Ran 7 tests in 0.348s
2023-01-11T20:49:53.1368030Z 
2023-01-11T20:49:53.1368102Z OK (skipped=2)
2023-01-11T20:49:53.1368195Z 
2023-01-11T20:49:53.1368279Z Generating XML reports...
2023-01-11T20:49:53.1368778Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20230111204952.xml
2023-01-11T20:49:53.1369055Z 
2023-01-11T20:49:53.1369289Z ##[endgroup]
2023-01-11T20:49:53.1369751Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_checkpoint_wrapper_carw9pv0)
2023-01-11T20:49:53.1370029Z 
2023-01-11T20:49:53.1370232Z Running distributed/test_multi_threaded_pg ... [2023-01-11 20:49:53.136251]
2023-01-11T20:49:53.1370746Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_multi_threaded_pg.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:53.136627]
2023-01-11T20:49:55.0498297Z 
2023-01-11T20:49:55.0498900Z Expand the folded group to see the log file of distributed/test_multi_threaded_pg
2023-01-11T20:49:55.0500061Z ##[group]PRINTING LOG FILE of distributed/test_multi_threaded_pg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_multi_threaded_pg_ulaicc_m)
2023-01-11T20:49:55.0501114Z Test results will be stored in test-reports/python-unittest/distributed.test_multi_threaded_pg
2023-01-11T20:49:55.0501477Z 
2023-01-11T20:49:55.0501603Z Running tests...
2023-01-11T20:49:55.0502125Z ----------------------------------------------------------------------
2023-01-11T20:49:55.0502871Z   test_all_reduce (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:49:55.0503629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:49:55.0505047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:49:55.0505706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:49:55.0506563Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0507460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0508255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0510693Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0511094Z ok (0.265s)
2023-01-11T20:49:55.0511567Z   test_allgather (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:49:55.0512471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:49:55.0513026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:49:55.0513801Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0514414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:49:55.0515169Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0515911Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0516637Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0517028Z ok (0.015s)
2023-01-11T20:49:55.0517625Z   test_assert_equal_on_rank (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:49:55.0518233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:49:55.0518724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:49:55.0519408Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0520126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:49:55.0520850Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0521627Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0522390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0522809Z ok (0.013s)
2023-01-11T20:49:55.0523317Z   test_broadcast (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:49:55.0523934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:49:55.0524472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:49:55.0525015Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:49:55.0525732Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0526483Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0527248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0528007Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0528414Z ok (0.017s)
2023-01-11T20:49:55.0528964Z   test_broadcast_object_list (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:49:55.0529667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:49:55.0530206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:49:55.0531172Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0531882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:49:55.0532601Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0533305Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0534026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0534461Z 2 -> 4
2023-01-11T20:49:55.0534699Z 3 -> 4
2023-01-11T20:49:55.0534930Z 0 -> 4
2023-01-11T20:49:55.0535159Z 1 -> 4
2023-01-11T20:49:55.0535370Z ok (0.014s)
2023-01-11T20:49:55.0535845Z   test_reduce_scatter (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:49:55.0536638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:49:55.0537226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:49:55.0537803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:49:55.0538506Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0539023Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0539532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0540029Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0540308Z ok (0.013s)
2023-01-11T20:49:55.0540649Z   test_scatter (__main__.TestCollectivesWithBaseClass) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:49:55.0541078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:49:55.0541422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:49:55.0541907Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0542291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:49:55.0542773Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0543265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0543774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0544056Z ok (0.013s)
2023-01-11T20:49:55.0544394Z   test_broadcast_object_list (__main__.TestCollectivesWithWrapper) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:49:55.0544833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:49:55.0545187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:49:55.0545667Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0546039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:49:55.0546523Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0547094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0547605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0547871Z ok (0.014s)
2023-01-11T20:49:55.0548231Z   test_collective_error_on_rank_non_zero (__main__.TestCollectivesWithWrapper) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:49:55.0548680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:49:55.0549040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:49:55.0549380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:49:55.0549899Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0550411Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0550918Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0551408Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0551774Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2023-01-11T20:49:55.0552034Z Traceback (most recent call last):
2023-01-11T20:49:55.0552436Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/multi_threaded_pg.py", line 365, in worker
2023-01-11T20:49:55.0552718Z     callback()
2023-01-11T20:49:55.0553095Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_distributed.py", line 896, in <lambda>
2023-01-11T20:49:55.0553430Z     "runTest", timeout, world_size, lambda: func(self, *args, **kwargs)
2023-01-11T20:49:55.0553706Z   File "distributed/test_multi_threaded_pg.py", line 57, in _test_method
2023-01-11T20:49:55.0553999Z     raise AssertionError("Mimic real test failure.")  # fail on rank 1
2023-01-11T20:49:55.0554257Z AssertionError: Mimic real test failure.
2023-01-11T20:49:55.0554449Z  exiting thread 1
2023-01-11T20:49:55.0554618Z ok (0.013s)
2023-01-11T20:49:55.0554987Z   test_collective_error_on_rank_non_zero_all (__main__.TestCollectivesWithWrapper) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:49:55.0555433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:49:55.0555796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:49:55.0556289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0556684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:49:55.0557156Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0557663Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0558170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0558526Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2023-01-11T20:49:55.0558770Z Traceback (most recent call last):
2023-01-11T20:49:55.0559178Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/multi_threaded_pg.py", line 365, in worker
2023-01-11T20:49:55.0559498Z     callback()
2023-01-11T20:49:55.0559854Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_distributed.py", line 896, in <lambda>
2023-01-11T20:49:55.0560187Z     "runTest", timeout, world_size, lambda: func(self, *args, **kwargs)
2023-01-11T20:49:55.0560475Z   File "distributed/test_multi_threaded_pg.py", line 72, in _test_method
2023-01-11T20:49:55.0560842Z     raise AssertionError("Mimic real test failure.")  # fail on all non-zero rank
2023-01-11T20:49:55.0561133Z AssertionError: Mimic real test failure.
2023-01-11T20:49:55.0561483Z  exiting thread 1
2023-01-11T20:49:55.0561762Z ok (0.012s)
2023-01-11T20:49:55.0562143Z   test_collective_error_on_rank_zero (__main__.TestCollectivesWithWrapper) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:49:55.0562586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:49:55.0562986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:49:55.0563483Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0563865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:49:55.0564347Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0564848Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0565358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0565701Z ERROR:torch.testing._internal.common_distributed:Caught exception: 
2023-01-11T20:49:55.0565959Z Traceback (most recent call last):
2023-01-11T20:49:55.0566366Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/distributed/multi_threaded_pg.py", line 365, in worker
2023-01-11T20:49:55.0566631Z     callback()
2023-01-11T20:49:55.0566997Z   File "/opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_distributed.py", line 896, in <lambda>
2023-01-11T20:49:55.0567323Z     "runTest", timeout, world_size, lambda: func(self, *args, **kwargs)
2023-01-11T20:49:55.0567605Z   File "distributed/test_multi_threaded_pg.py", line 42, in _test_method
2023-01-11T20:49:55.0567879Z     raise AssertionError("Mimic real test failure.")  # fail on rank 0
2023-01-11T20:49:55.0568135Z AssertionError: Mimic real test failure.
2023-01-11T20:49:55.0568334Z  exiting thread 0
2023-01-11T20:49:55.0568489Z ok (0.012s)
2023-01-11T20:49:55.0568828Z   test_skip (__main__.TestCollectivesWithWrapper) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:49:55.0569255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:49:55.0569607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:49:55.0570094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0570486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:49:55.0571130Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0571637Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0572147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:49:55.0572595Z INFO:torch.testing._internal.common_distributed:Thread 1 skipping test runTest for following reason: check if skip exception can be captured correctly.
2023-01-11T20:49:55.0573135Z INFO:torch.testing._internal.common_distributed:Thread 2 skipping test runTest for following reason: check if skip exception can be captured correctly.
2023-01-11T20:49:55.0573585Z INFO:torch.testing._internal.common_distributed:Thread 3 skipping test runTest for following reason: check if skip exception can be captured correctly.
2023-01-11T20:49:55.0574041Z INFO:torch.testing._internal.common_distributed:Thread 0 skipping test runTest for following reason: check if skip exception can be captured correctly.
2023-01-11T20:49:55.0574340Z ok (0.011s)
2023-01-11T20:49:55.0574442Z 
2023-01-11T20:49:55.0574647Z ----------------------------------------------------------------------
2023-01-11T20:49:55.0574874Z Ran 12 tests in 0.413s
2023-01-11T20:49:55.0574987Z 
2023-01-11T20:49:55.0575049Z OK
2023-01-11T20:49:55.0575140Z 
2023-01-11T20:49:55.0575225Z Generating XML reports...
2023-01-11T20:49:55.0575738Z Generated XML report: test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithBaseClass-20230111204954.xml
2023-01-11T20:49:55.0576435Z Generated XML report: test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithWrapper-20230111204954.xml
2023-01-11T20:49:55.0576717Z 
2023-01-11T20:49:55.0577022Z ##[endgroup]
2023-01-11T20:49:55.0577469Z FINISHED PRINTING LOG FILE of distributed/test_multi_threaded_pg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_multi_threaded_pg_ulaicc_m)
2023-01-11T20:49:55.0577709Z 
2023-01-11T20:49:55.0577910Z Running distributed/test_dynamo_distributed ... [2023-01-11 20:49:55.050372]
2023-01-11T20:49:55.0578431Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:55.050803]
2023-01-11T20:49:57.2263065Z 
2023-01-11T20:49:57.2263543Z Expand the folded group to see the log file of distributed/test_dynamo_distributed
2023-01-11T20:49:57.2264311Z ##[group]PRINTING LOG FILE of distributed/test_dynamo_distributed (/var/lib/jenkins/workspace/test/test-reports/distributed-test_dynamo_distributed_eo9axzjq)
2023-01-11T20:49:57.2264606Z 
2023-01-11T20:49:57.2264723Z Running tests...
2023-01-11T20:49:57.2265128Z ----------------------------------------------------------------------
2023-01-11T20:49:57.2265391Z   test_aot_autograd (__main__.TestDistributed)
2023-01-11T20:49:57.2265930Z Explicitly check AotAutograd family of compilers work, ... Test results will be stored in test-reports/python-unittest/distributed.test_dynamo_distributed
2023-01-11T20:49:57.2266356Z skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:57.2266603Z   test_custom_layer (__main__.TestDistributed)
2023-01-11T20:49:57.2267112Z Just ensures that the appropriate number of splits happen (based on ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:57.2267538Z   test_ddp_baseline_aot_eager (__main__.TestDistributed) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:49:57.2268007Z   test_ddp_baseline_inductor (__main__.TestDistributed) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:49:57.2268371Z   test_empty_graph_inductor (__main__.TestDistributed) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:49:57.2268806Z   test_fsdp_orig_params_assert (__main__.TestDistributed) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:49:57.2269108Z   test_graph_split (__main__.TestDistributed)
2023-01-11T20:49:57.2269498Z Just ensures that the appropriate number of splits happen (based on ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:57.2269813Z   test_graph_split_inductor (__main__.TestDistributed)
2023-01-11T20:49:57.2270144Z Same as above, but using inductor backend. ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:49:57.2270681Z   test_ignored_parameters (__main__.TestDistributed)
2023-01-11T20:49:57.2271281Z Verifies ddp graph-split logic ignores parameters marked to ignore on DDP module. ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:57.2271613Z   test_no_split (__main__.TestDistributed)
2023-01-11T20:49:57.2271935Z Ensures the DDPOptimizer returns a correct, compiled module without ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:57.2272364Z   test_ddp_baseline_aot_eager_multiprocess (__main__.TestDistributedMultiProc) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:49:57.2272768Z   test_fsdp_aot_eager (__main__.TestDistributedMultiProc) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:57.2273165Z   test_fsdp_inductor (__main__.TestDistributedMultiProc) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:57.2273672Z   test_hf_bert_ddp_aot_eager (__main__.TestDistributedMultiProc) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:49:57.2274086Z   test_hf_bert_ddp_inductor (__main__.TestDistributedMultiProc) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:49:57.2274466Z   test_hf_bert_fsdp (__main__.TestDistributedMultiProc) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:49:57.2274870Z   test_hf_bert_ddp_aot_eager (__main__.TestFakeDistributedSingleProc) ... skip: Unable to import transformers (0.248s)
2023-01-11T20:49:57.2275298Z   test_hf_bert_ddp_inductor (__main__.TestFakeDistributedSingleProc) ... skip: Inductor+gpu needs triton and recent GPU arch (0.000s)
2023-01-11T20:49:57.2275775Z   test_issue90375 (__main__.TestFakeDistributedSingleProc) ... frames [('total', 4), ('ok', 3)]
2023-01-11T20:49:57.2276170Z inline_call [('inline in skipfiles: helper /opt/conda/lib/python3.7/contextlib.py', 1)]
2023-01-11T20:49:57.2276616Z unimplemented [('SETUP_WITH UserDefinedObjectVariable(_GeneratorContextManager)', 1), ('generator', 1)]
2023-01-11T20:49:57.2277051Z graph_break [('inline in skipfiles: helper /opt/conda/lib/python3.7/contextlib.py', 1)]
2023-01-11T20:49:57.2277422Z stats [('calls_captured', 3), ('fusions_possible', 2), ('unique_graphs', 1)]
2023-01-11T20:49:57.2277696Z aot_autograd [('total', 1), ('ok', 1)]
2023-01-11T20:49:57.2277879Z ok (0.227s)
2023-01-11T20:49:57.2277981Z 
2023-01-11T20:49:57.2278178Z ----------------------------------------------------------------------
2023-01-11T20:49:57.2278405Z Ran 19 tests in 0.484s
2023-01-11T20:49:57.2278517Z 
2023-01-11T20:49:57.2278588Z OK (skipped=18)
2023-01-11T20:49:57.2278692Z 
2023-01-11T20:49:57.2278775Z Generating XML reports...
2023-01-11T20:49:57.2279257Z Generated XML report: test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestFakeDistributedSingleProc-20230111204956.xml
2023-01-11T20:49:57.2279857Z Generated XML report: test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributed-20230111204956.xml
2023-01-11T20:49:57.2280444Z Generated XML report: test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributedMultiProc-20230111204956.xml
2023-01-11T20:49:57.2280717Z 
2023-01-11T20:49:57.2280977Z ##[endgroup]
2023-01-11T20:49:57.2281413Z FINISHED PRINTING LOG FILE of distributed/test_dynamo_distributed (/var/lib/jenkins/workspace/test/test-reports/distributed-test_dynamo_distributed_eo9axzjq)
2023-01-11T20:49:57.2281672Z 
2023-01-11T20:49:57.2281883Z Running distributed/fsdp/test_fsdp_traversal ... [2023-01-11 20:49:57.226615]
2023-01-11T20:49:57.2282399Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:49:57.226962]
2023-01-11T20:50:00.6817543Z 
2023-01-11T20:50:00.6818141Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_traversal
2023-01-11T20:50:00.6819108Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_traversal_z_s8_rzu)
2023-01-11T20:50:00.6819983Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal
2023-01-11T20:50:00.6820191Z 
2023-01-11T20:50:00.6820251Z Running tests...
2023-01-11T20:50:00.6820558Z ----------------------------------------------------------------------
2023-01-11T20:50:00.6820934Z   test_fsdp_modules (__main__.TestTraversal) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7334
2023-01-11T20:50:00.6821366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7335
2023-01-11T20:50:00.6821822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:00.6822209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:00.6822727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:00.6823226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:00.6823576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:50:00.6824086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:00.6824467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:00.6824885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:00.6825319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:00.6825715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:50:00.6826351Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:50:00.6826919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:50:00.6827308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:50:00.6827652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:50:00.6827915Z dist init r=0, world=2
2023-01-11T20:50:00.6828086Z dist init r=1, world=2
2023-01-11T20:50:00.6828294Z skip: Need at least 2 CUDA devices (1.866s)
2023-01-11T20:50:00.6828431Z 
2023-01-11T20:50:00.6828630Z ----------------------------------------------------------------------
2023-01-11T20:50:00.6828859Z Ran 1 test in 1.866s
2023-01-11T20:50:00.6828971Z 
2023-01-11T20:50:00.6829044Z OK (skipped=1)
2023-01-11T20:50:00.6829151Z 
2023-01-11T20:50:00.6829238Z Generating XML reports...
2023-01-11T20:50:00.6829677Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20230111204958.xml
2023-01-11T20:50:00.6829935Z 
2023-01-11T20:50:00.6830189Z ##[endgroup]
2023-01-11T20:50:00.6830656Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_traversal (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_traversal_z_s8_rzu)
2023-01-11T20:50:00.6830920Z 
2023-01-11T20:50:00.6831164Z Running distributed/tensor/parallel/test_view_sharding_dim_change ... [2023-01-11 20:50:00.682083]
2023-01-11T20:50:00.6831722Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/tensor/parallel/test_view_sharding_dim_change.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:50:00.682470]
2023-01-11T20:50:04.3219245Z 
2023-01-11T20:50:04.3220012Z Expand the folded group to see the log file of distributed/tensor/parallel/test_view_sharding_dim_change
2023-01-11T20:50:04.3221195Z ##[group]PRINTING LOG FILE of distributed/tensor/parallel/test_view_sharding_dim_change (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_view_sharding_dim_change_lsmcj5kh)
2023-01-11T20:50:04.3222484Z Test results will be stored in test-reports/python-unittest/distributed.tensor.parallel.test_view_sharding_dim_change
2023-01-11T20:50:04.3222721Z 
2023-01-11T20:50:04.3222811Z Running tests...
2023-01-11T20:50:04.3223119Z ----------------------------------------------------------------------
2023-01-11T20:50:04.3223562Z   test_view_with_sharding_dim_change (__main__.TPViewShardingDimChangeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7373
2023-01-11T20:50:04.3224166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7374
2023-01-11T20:50:04.3224501Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7375
2023-01-11T20:50:04.3224974Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7376
2023-01-11T20:50:04.3225547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:04.3225987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:04.3226422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:04.3226757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:04.3227082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:50:04.3227540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:04.3227956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:04.3228539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:04.3228884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:04.3229258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:50:04.3229712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:04.3230033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:04.3230457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:04.3230796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:04.3231101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:50:04.3231558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:04.3231900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:04.3232322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:04.3232653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:04.3232966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:50:04.3233328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:50:04.3233686Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:50:04.3234175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:04.3234566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:50:04.3234924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:50:04.3235398Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:04.3235976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:04.3236486Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:04.3236772Z ok (2.071s)
2023-01-11T20:50:04.3236863Z 
2023-01-11T20:50:04.3237062Z ----------------------------------------------------------------------
2023-01-11T20:50:04.3237302Z Ran 1 test in 2.071s
2023-01-11T20:50:04.3237417Z 
2023-01-11T20:50:04.3237478Z OK
2023-01-11T20:50:04.3237569Z 
2023-01-11T20:50:04.3237641Z Generating XML reports...
2023-01-11T20:50:04.3238173Z Generated XML report: test-reports/python-unittest/distributed.tensor.parallel.test_view_sharding_dim_change/TEST-TPViewShardingDimChangeTest-20230111205001.xml
2023-01-11T20:50:04.3238493Z 
2023-01-11T20:50:04.3238756Z ##[endgroup]
2023-01-11T20:50:04.3239329Z FINISHED PRINTING LOG FILE of distributed/tensor/parallel/test_view_sharding_dim_change (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_view_sharding_dim_change_lsmcj5kh)
2023-01-11T20:50:04.3239648Z 
2023-01-11T20:50:04.3239843Z Running distributed/rpc/test_share_memory ... [2023-01-11 20:50:04.322262]
2023-01-11T20:50:04.3240380Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_share_memory.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:50:04.322707]
2023-01-11T20:50:09.3047764Z 
2023-01-11T20:50:09.3048240Z Expand the folded group to see the log file of distributed/rpc/test_share_memory
2023-01-11T20:50:09.3049237Z ##[group]PRINTING LOG FILE of distributed/rpc/test_share_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_share_memory_ctcdld93)
2023-01-11T20:50:09.3049880Z <unittest.suite.TestSuite tests=[]>
2023-01-11T20:50:09.3050345Z <unittest.suite.TestSuite tests=[<__main__.TestRPCPickler testMethod=test_case>]>
2023-01-11T20:50:09.3050769Z test_case (__main__.TestRPCPickler)
2023-01-11T20:50:09.3051762Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_share_memory
2023-01-11T20:50:09.3052635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:09.3052970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:09.3053511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:09.3053843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:09.3054008Z 
2023-01-11T20:50:09.3054082Z Running tests...
2023-01-11T20:50:09.3054388Z ----------------------------------------------------------------------
2023-01-11T20:50:09.3054890Z   test_case (__main__.TestRPCPickler) ... /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:09.3055247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:09.3055671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:09.3056016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:09.3056390Z Exception ignored in: <function StorageWeakRef.__del__ at 0x7fda08013200>
2023-01-11T20:50:09.3056673Z Traceback (most recent call last):
2023-01-11T20:50:09.3057147Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 38, in __del__
2023-01-11T20:50:09.3057571Z   File "/opt/conda/lib/python3.7/site-packages/torch/storage.py", line 768, in _free_weak_ref
2023-01-11T20:50:09.3057925Z AttributeError: 'NoneType' object has no attribute '_free_weak_ref'
2023-01-11T20:50:09.3058221Z Exception ignored in: <function StorageWeakRef.__del__ at 0x7fda08013200>
2023-01-11T20:50:09.3058476Z Traceback (most recent call last):
2023-01-11T20:50:09.3059050Z   File "/opt/conda/lib/python3.7/site-packages/torch/multiprocessing/reductions.py", line 38, in __del__
2023-01-11T20:50:09.3059470Z   File "/opt/conda/lib/python3.7/site-packages/torch/storage.py", line 768, in _free_weak_ref
2023-01-11T20:50:09.3059834Z AttributeError: 'NoneType' object has no attribute '_free_weak_ref'
2023-01-11T20:50:09.3060665Z /opt/conda/lib/python3.7/site-packages/torch/multiprocessing/reductions.py:355: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T20:50:09.3061193Z   if storage.is_cuda:
2023-01-11T20:50:09.3061353Z ok (2.042s)
2023-01-11T20:50:09.3061457Z 
2023-01-11T20:50:09.3061653Z ----------------------------------------------------------------------
2023-01-11T20:50:09.3061887Z Ran 1 test in 2.043s
2023-01-11T20:50:09.3062082Z 
2023-01-11T20:50:09.3062131Z OK
2023-01-11T20:50:09.3062222Z 
2023-01-11T20:50:09.3062305Z Generating XML reports...
2023-01-11T20:50:09.3062746Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20230111205006.xml
2023-01-11T20:50:09.3063000Z 
2023-01-11T20:50:09.3063248Z ##[endgroup]
2023-01-11T20:50:09.3063692Z FINISHED PRINTING LOG FILE of distributed/rpc/test_share_memory (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_share_memory_ctcdld93)
2023-01-11T20:50:09.3063943Z 
2023-01-11T20:50:09.3064156Z Running distributed/fsdp/test_fsdp_multiple_forward ... [2023-01-11 20:50:09.305206]
2023-01-11T20:50:09.3064698Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:50:09.305589]
2023-01-11T20:50:12.9543486Z 
2023-01-11T20:50:12.9544211Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_multiple_forward
2023-01-11T20:50:12.9545461Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_forward (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_forward_p_v9xznq)
2023-01-11T20:50:12.9546603Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward
2023-01-11T20:50:12.9546988Z 
2023-01-11T20:50:12.9547099Z Running tests...
2023-01-11T20:50:12.9547636Z ----------------------------------------------------------------------
2023-01-11T20:50:12.9548325Z   test_multi_forward (__main__.TestMultiForward) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7557
2023-01-11T20:50:12.9548929Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7558
2023-01-11T20:50:12.9549455Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7559
2023-01-11T20:50:12.9550040Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7560
2023-01-11T20:50:12.9550878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:12.9551461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:12.9552260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:12.9552890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:12.9553491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:50:12.9554258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:12.9554586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:12.9555009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:12.9555564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:12.9555916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:50:12.9556387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:12.9556714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:12.9557124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:12.9557462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:12.9557792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:50:12.9558255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:12.9558633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:12.9559059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:12.9559401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:12.9559730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:50:12.9560208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:12.9560726Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:12.9561239Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:12.9561740Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:12.9562126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:50:12.9562470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:50:12.9562811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:50:12.9563138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:50:12.9563387Z dist init r=2, world=4
2023-01-11T20:50:12.9563567Z dist init r=0, world=4
2023-01-11T20:50:12.9563730Z dist init r=1, world=4
2023-01-11T20:50:12.9563907Z dist init r=3, world=4
2023-01-11T20:50:12.9564111Z skip: Need at least 2 CUDA devices (2.066s)
2023-01-11T20:50:12.9564246Z 
2023-01-11T20:50:12.9564444Z ----------------------------------------------------------------------
2023-01-11T20:50:12.9564671Z Ran 1 test in 2.066s
2023-01-11T20:50:12.9564782Z 
2023-01-11T20:50:12.9564857Z OK (skipped=1)
2023-01-11T20:50:12.9564965Z 
2023-01-11T20:50:12.9565049Z Generating XML reports...
2023-01-11T20:50:12.9565498Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20230111205010.xml
2023-01-11T20:50:12.9565764Z 
2023-01-11T20:50:12.9566013Z ##[endgroup]
2023-01-11T20:50:12.9566493Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_multiple_forward (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_multiple_forward_p_v9xznq)
2023-01-11T20:50:12.9566769Z 
2023-01-11T20:50:12.9566965Z Running distributed/elastic/utils/distributed_test ... [2023-01-11 20:50:12.954734]
2023-01-11T20:50:12.9567502Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/distributed_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:50:12.955167]
2023-01-11T20:50:17.8587518Z 
2023-01-11T20:50:17.8588086Z Expand the folded group to see the log file of distributed/elastic/utils/distributed_test
2023-01-11T20:50:17.8589102Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/distributed_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-distributed_test_h7m1ztzo)
2023-01-11T20:50:17.8589862Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.distributed_test
2023-01-11T20:50:17.8590095Z 
2023-01-11T20:50:17.8590199Z Running tests...
2023-01-11T20:50:17.8590517Z ----------------------------------------------------------------------
2023-01-11T20:50:17.8590842Z   test_create_store_multi (__main__.DistributedUtilTest) ... ok (0.258s)
2023-01-11T20:50:17.8591226Z   test_create_store_no_port_multi (__main__.DistributedUtilTest) ... ok (0.001s)
2023-01-11T20:50:17.8591523Z   test_create_store_single_server (__main__.DistributedUtilTest) ... ok (0.003s)
2023-01-11T20:50:17.8591914Z   test_create_store_timeout_on_server (__main__.DistributedUtilTest) ... ok (3.023s)
2023-01-11T20:50:17.8592512Z   test_create_store_timeout_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (16a6623252a2, 0).
2023-01-11T20:50:17.8592818Z ok (0.001s)
2023-01-11T20:50:17.8593308Z   test_port_already_in_use_on_server (__main__.DistributedUtilTest) ... [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:43483 (errno: 98 - Address already in use).
2023-01-11T20:50:17.8593816Z [W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:43483 (errno: 98 - Address already in use).
2023-01-11T20:50:17.8594154Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address.
2023-01-11T20:50:17.8594446Z ok (0.004s)
2023-01-11T20:50:17.8594776Z   test_port_already_in_use_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (16a6623252a2, 36995).
2023-01-11T20:50:17.8595109Z ok (0.001s)
2023-01-11T20:50:17.8595267Z 
2023-01-11T20:50:17.8595472Z ----------------------------------------------------------------------
2023-01-11T20:50:17.8595696Z Ran 7 tests in 3.292s
2023-01-11T20:50:17.8595808Z 
2023-01-11T20:50:17.8595868Z OK
2023-01-11T20:50:17.8595957Z 
2023-01-11T20:50:17.8596039Z Generating XML reports...
2023-01-11T20:50:17.8596517Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20230111205014.xml
2023-01-11T20:50:17.8596804Z 
2023-01-11T20:50:17.8597031Z ##[endgroup]
2023-01-11T20:50:17.8597523Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/distributed_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-distributed_test_h7m1ztzo)
2023-01-11T20:50:17.8597810Z 
2023-01-11T20:50:17.8598033Z Running distributed/fsdp/test_distributed_checkpoint ... [2023-01-11 20:50:17.859050]
2023-01-11T20:50:17.8598563Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:50:17.859394]
2023-01-11T20:50:22.9528238Z 
2023-01-11T20:50:22.9528864Z Expand the folded group to see the log file of distributed/fsdp/test_distributed_checkpoint
2023-01-11T20:50:22.9530278Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_distributed_checkpoint_4yu5j6nq)
2023-01-11T20:50:22.9531395Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint
2023-01-11T20:50:22.9531616Z 
2023-01-11T20:50:22.9531691Z Running tests...
2023-01-11T20:50:22.9532062Z ----------------------------------------------------------------------
2023-01-11T20:50:22.9532703Z   test_distributed_checkpoint_state_dict_type_StateDictType_LOCAL_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7646
2023-01-11T20:50:22.9533270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7647
2023-01-11T20:50:22.9534269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:22.9534589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:22.9535017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:22.9535431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:22.9535917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:50:22.9536565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:22.9537173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:22.9537732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:22.9538068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:22.9538400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:50:22.9538896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:50:22.9539421Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:50:22.9539792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:50:22.9540138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:50:22.9540400Z dist init r=1, world=2
2023-01-11T20:50:22.9540565Z dist init r=0, world=2
2023-01-11T20:50:22.9540773Z skip: Need at least 2 CUDA devices (1.869s)
2023-01-11T20:50:22.9541195Z   test_distributed_checkpoint_state_dict_type_StateDictType_SHARDED_STATE_DICT (__main__.TestDistributedCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7674
2023-01-11T20:50:22.9541648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7675
2023-01-11T20:50:22.9542084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:22.9542407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:22.9542830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:22.9543157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:22.9543487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:50:22.9543955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:22.9544280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:22.9544687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:22.9545027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:22.9545354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:50:22.9545826Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:50:22.9546340Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:50:22.9546719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:50:22.9547061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:50:22.9547355Z dist init r=1, world=2
2023-01-11T20:50:22.9547534Z dist init r=0, world=2
2023-01-11T20:50:22.9547740Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:50:22.9547874Z 
2023-01-11T20:50:22.9548061Z ----------------------------------------------------------------------
2023-01-11T20:50:22.9548302Z Ran 2 tests in 3.482s
2023-01-11T20:50:22.9548413Z 
2023-01-11T20:50:22.9548485Z OK (skipped=2)
2023-01-11T20:50:22.9548590Z 
2023-01-11T20:50:22.9548674Z Generating XML reports...
2023-01-11T20:50:22.9549157Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20230111205019.xml
2023-01-11T20:50:22.9549450Z 
2023-01-11T20:50:22.9549736Z ##[endgroup]
2023-01-11T20:50:22.9550229Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_distributed_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_distributed_checkpoint_4yu5j6nq)
2023-01-11T20:50:22.9550515Z 
2023-01-11T20:50:22.9550764Z Running distributed/checkpoint/test_checkpoint ... [2023-01-11 20:50:22.953139]
2023-01-11T20:50:22.9551300Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:50:22.953489]
2023-01-11T20:50:28.2466824Z 
2023-01-11T20:50:28.2467438Z Expand the folded group to see the log file of distributed/checkpoint/test_checkpoint
2023-01-11T20:50:28.2468659Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_checkpoint_s_47wdyg)
2023-01-11T20:50:28.2469641Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_checkpoint
2023-01-11T20:50:28.2469978Z 
2023-01-11T20:50:28.2470082Z Running tests...
2023-01-11T20:50:28.2470641Z ----------------------------------------------------------------------
2023-01-11T20:50:28.2471288Z   test_default_metadata (__main__.TestDistributedCheckpointing) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:50:28.2471857Z   test_tensor_metadata_with_missing_rank_spec (__main__.TestDistributedCheckpointing) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:50:28.2472321Z   test_dummy_reader_works (__main__.TestDistributedFailure) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:50:28.2472990Z   test_dummy_writer_works (__main__.TestDistributedFailure) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T20:50:28.2473639Z   test_load_error_handling (__main__.TestDistributedFailure) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:50:28.2474376Z   test_load_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7713
2023-01-11T20:50:28.2474968Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7714
2023-01-11T20:50:28.2475299Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7715
2023-01-11T20:50:28.2475607Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7716
2023-01-11T20:50:28.2476083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:28.2476433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:28.2476868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:28.2494489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:28.2494980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:50:28.2495615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:28.2495980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:28.2496715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:28.2497051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:28.2497366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:50:28.2497825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:28.2498140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:28.2498565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:28.2498907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:28.2499212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:50:28.2499798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:28.2500275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:28.2500845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:28.2501443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:28.2501762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:50:28.2502008Z ok (1.971s)
2023-01-11T20:50:28.2502283Z   test_save_error_handling (__main__.TestDistributedFailure) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T20:50:28.2502722Z   test_save_error_handling_no_dist (__main__.TestDistributedFailure) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7757
2023-01-11T20:50:28.2503116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7758
2023-01-11T20:50:28.2503476Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7759
2023-01-11T20:50:28.2503997Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7760
2023-01-11T20:50:28.2504703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:28.2505030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:28.2505454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:28.2505779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:28.2506094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:50:28.2506545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:28.2506860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:28.2507283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:28.2507622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:28.2507937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:50:28.2508380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:28.2508703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:28.2509124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:28.2509460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:28.2509764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:50:28.2510290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:28.2510613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:28.2511021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:28.2511357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:28.2511677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:50:28.2511932Z ok (1.719s)
2023-01-11T20:50:28.2512021Z 
2023-01-11T20:50:28.2512218Z ----------------------------------------------------------------------
2023-01-11T20:50:28.2512455Z Ran 8 tests in 3.695s
2023-01-11T20:50:28.2512565Z 
2023-01-11T20:50:28.2512635Z OK (skipped=6)
2023-01-11T20:50:28.2512738Z 
2023-01-11T20:50:28.2512809Z Generating XML reports...
2023-01-11T20:50:28.2513327Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20230111205024.xml
2023-01-11T20:50:28.2513977Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20230111205024.xml
2023-01-11T20:50:28.2514275Z 
2023-01-11T20:50:28.2514569Z ##[endgroup]
2023-01-11T20:50:28.2515040Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_checkpoint_s_47wdyg)
2023-01-11T20:50:28.2515321Z 
2023-01-11T20:50:28.2515524Z Running distributed/fsdp/test_fsdp_pure_fp16 ... [2023-01-11 20:50:28.247175]
2023-01-11T20:50:28.2516042Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_pure_fp16.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:50:28.247590]
2023-01-11T20:50:33.6155537Z 
2023-01-11T20:50:33.6156136Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_pure_fp16
2023-01-11T20:50:33.6157321Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_pure_fp16 (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_pure_fp16_gw_ancrl)
2023-01-11T20:50:33.6158384Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16
2023-01-11T20:50:33.6158749Z 
2023-01-11T20:50:33.6158872Z Running tests...
2023-01-11T20:50:33.6159417Z ----------------------------------------------------------------------
2023-01-11T20:50:33.6159854Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=False) (__main__.TestPureFP16)
2023-01-11T20:50:33.6160374Z Tests pure FP16 training, including when the parameter's dtype is ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7812
2023-01-11T20:50:33.6160781Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7813
2023-01-11T20:50:33.6161226Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7814
2023-01-11T20:50:33.6161664Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7815
2023-01-11T20:50:33.6162297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:33.6162826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:33.6163638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:33.6164270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:33.6164883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:50:33.6165724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:33.6166192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:33.6166868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:33.6167607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:33.6167943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:50:33.6168413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:33.6168737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:33.6169146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:33.6169493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:33.6169825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:50:33.6170363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:33.6170679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:33.6171274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:33.6171614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:33.6171933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:50:33.6172431Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:33.6172946Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:33.6173463Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:33.6173962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:33.6174348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:50:33.6174693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:50:33.6175034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:50:33.6175360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:50:33.6175614Z dist init r=1, world=4
2023-01-11T20:50:33.6175796Z dist init r=0, world=4
2023-01-11T20:50:33.6175958Z dist init r=3, world=4
2023-01-11T20:50:33.6176135Z dist init r=2, world=4
2023-01-11T20:50:33.6176402Z skip: Need at least 2 CUDA devices (1.972s)
2023-01-11T20:50:33.6176680Z   test_pure_fp16_cpu_offload_CPUOffload(offload_params=True) (__main__.TestPureFP16)
2023-01-11T20:50:33.6177200Z Tests pure FP16 training, including when the parameter's dtype is ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7868
2023-01-11T20:50:33.6177597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7869
2023-01-11T20:50:33.6177918Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7870
2023-01-11T20:50:33.6178222Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7871
2023-01-11T20:50:33.6178665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:33.6178997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:33.6179407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:33.6179750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:33.6180087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:50:33.6180650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:33.6180978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:33.6181387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:33.6181727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:33.6182057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:50:33.6182509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:33.6182831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:33.6183299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:33.6183643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:33.6183959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:50:33.6184421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:33.6184743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:33.6185154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:33.6185492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:33.6185818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:50:33.6186310Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:33.6186817Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:33.6187332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:33.6187845Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:33.6188220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:50:33.6188550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:50:33.6188884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:50:33.6189225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:50:33.6189472Z dist init r=2, world=4
2023-01-11T20:50:33.6189655Z dist init r=0, world=4
2023-01-11T20:50:33.6189829Z dist init r=3, world=4
2023-01-11T20:50:33.6190002Z dist init r=1, world=4
2023-01-11T20:50:33.6190191Z skip: Need at least 2 CUDA devices (1.819s)
2023-01-11T20:50:33.6190326Z 
2023-01-11T20:50:33.6190523Z ----------------------------------------------------------------------
2023-01-11T20:50:33.6190758Z Ran 2 tests in 3.791s
2023-01-11T20:50:33.6190858Z 
2023-01-11T20:50:33.6190927Z OK (skipped=2)
2023-01-11T20:50:33.6191034Z 
2023-01-11T20:50:33.6191119Z Generating XML reports...
2023-01-11T20:50:33.6191559Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20230111205029.xml
2023-01-11T20:50:33.6191810Z 
2023-01-11T20:50:33.6192084Z ##[endgroup]
2023-01-11T20:50:33.6192535Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_pure_fp16 (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_pure_fp16_gw_ancrl)
2023-01-11T20:50:33.6192794Z 
2023-01-11T20:50:33.6193060Z Running distributed/checkpoint/test_fsdp_model_state ... [2023-01-11 20:50:33.616033]
2023-01-11T20:50:33.6193597Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_model_state.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:50:33.616482]
2023-01-11T20:50:38.9916285Z 
2023-01-11T20:50:38.9916907Z Expand the folded group to see the log file of distributed/checkpoint/test_fsdp_model_state
2023-01-11T20:50:38.9918069Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_fsdp_model_state (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_fsdp_model_state__m75xqy8)
2023-01-11T20:50:38.9919147Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_fsdp_model_state
2023-01-11T20:50:38.9919482Z 
2023-01-11T20:50:38.9919597Z Running tests...
2023-01-11T20:50:38.9920006Z ----------------------------------------------------------------------
2023-01-11T20:50:38.9921075Z   test_fsdp_model_state_no_resharding (__main__.FsdpModelStateCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7935
2023-01-11T20:50:38.9922029Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7936
2023-01-11T20:50:38.9922770Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7937
2023-01-11T20:50:38.9923474Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7938
2023-01-11T20:50:38.9924533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:38.9925269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:38.9926238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:38.9926984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:38.9927709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:50:38.9928745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:38.9929476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:38.9930410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:38.9931405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:38.9932114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:50:38.9933119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:38.9933834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:38.9934790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:38.9935549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:38.9936235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:50:38.9937327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:38.9938062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:38.9938988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:38.9939747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:38.9940467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:50:38.9941275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:50:38.9942076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:50:38.9943052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:50:38.9944150Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:38.9945031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:50:38.9946096Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:38.9947244Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:38.9948397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:38.9949100Z skip: Need at least 4 CUDA devices (2.070s)
2023-01-11T20:50:38.9949989Z   test_fsdp_model_state_with_resharding (__main__.FsdpModelStateCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7991
2023-01-11T20:50:38.9950918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7992
2023-01-11T20:50:38.9951648Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 7993
2023-01-11T20:50:38.9952348Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 7994
2023-01-11T20:50:38.9953352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:38.9954068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:38.9955019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:38.9955760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:38.9956473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:50:38.9957500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:38.9958216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:38.9959022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:38.9959518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:38.9959834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:50:38.9960284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:38.9960620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:38.9961040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:38.9961380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:38.9961685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:50:38.9962133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:38.9962452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:38.9962873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:38.9963201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:38.9963519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:50:38.9963876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:50:38.9964291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:50:38.9964643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:50:38.9965000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:50:38.9965489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:38.9965992Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:38.9966504Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:38.9967016Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:38.9967375Z skip: Need at least 4 CUDA devices (1.719s)
2023-01-11T20:50:38.9967510Z 
2023-01-11T20:50:38.9967697Z ----------------------------------------------------------------------
2023-01-11T20:50:38.9967936Z Ran 2 tests in 3.789s
2023-01-11T20:50:38.9968047Z 
2023-01-11T20:50:38.9968117Z OK (skipped=2)
2023-01-11T20:50:38.9968222Z 
2023-01-11T20:50:38.9968306Z Generating XML reports...
2023-01-11T20:50:38.9968784Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_fsdp_model_state/TEST-FsdpModelStateCheckpoint-20230111205034.xml
2023-01-11T20:50:38.9969077Z 
2023-01-11T20:50:38.9969363Z ##[endgroup]
2023-01-11T20:50:38.9969857Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_fsdp_model_state (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_fsdp_model_state__m75xqy8)
2023-01-11T20:50:38.9970143Z 
2023-01-11T20:50:38.9970340Z Running distributed/fsdp/test_fsdp_clip_grad_norm ... [2023-01-11 20:50:38.992037]
2023-01-11T20:50:38.9971040Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:50:38.992497]
2023-01-11T20:50:46.0851888Z 
2023-01-11T20:50:46.0875409Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_clip_grad_norm
2023-01-11T20:50:46.0876584Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_clip_grad_norm_s8_fmiuz)
2023-01-11T20:50:46.0877482Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm
2023-01-11T20:50:46.0877829Z 
2023-01-11T20:50:46.0877950Z Running tests...
2023-01-11T20:50:46.0878459Z ----------------------------------------------------------------------
2023-01-11T20:50:46.0878939Z   test_ddp_parity (__main__.TestClipGradNorm)
2023-01-11T20:50:46.0879625Z Tests FSDP with ``FullyShardedDataParallel.clip_grad_norm_()` against ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8058
2023-01-11T20:50:46.0880285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8059
2023-01-11T20:50:46.0880825Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8060
2023-01-11T20:50:46.0881360Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8061
2023-01-11T20:50:46.0882120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:46.0882635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:46.0883333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:46.0883877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:46.0884351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:50:46.0885079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:46.0885532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:46.0885990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:46.0886316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:46.0886647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:50:46.0887141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:46.0887630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:46.0887936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:46.0888433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:46.0888781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:46.0889097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:50:46.0889585Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:46.0890084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:46.0890406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:46.0890819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:46.0891356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:46.0891688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:50:46.0892172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:46.0892689Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:46.0893068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:50:46.0893411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:50:46.0893738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:50:46.0894079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:50:46.0894330Z dist init r=0, world=4
2023-01-11T20:50:46.0894507Z dist init r=3, world=4
2023-01-11T20:50:46.0894668Z dist init r=1, world=4
2023-01-11T20:50:46.0894846Z dist init r=2, world=4
2023-01-11T20:50:46.0895049Z skip: Need at least 2 CUDA devices (2.067s)
2023-01-11T20:50:46.0895283Z   test_low_precision_grads (__main__.TestClipGradNorm)
2023-01-11T20:50:46.0895652Z Tests ``clip_grad_norm_()`` when using low precision gradients. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8114
2023-01-11T20:50:46.0896032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8115
2023-01-11T20:50:46.0896399Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8116
2023-01-11T20:50:46.0896715Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8117
2023-01-11T20:50:46.0897163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:46.0897488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:46.0897954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:46.0898627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:46.0899057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:50:46.0899517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:46.0899840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:46.0900260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:46.0900597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:46.0900912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:50:46.0901430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:46.0901757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:46.0902176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:46.0902565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:46.0902879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:50:46.0903344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:46.0903663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:46.0904069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:46.0904404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:46.0904735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:50:46.0905224Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:46.0905724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:46.0906233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:46.0906740Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:46.0907118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:50:46.0907449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:50:46.0907783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:50:46.0908123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:50:46.0908365Z dist init r=3, world=4
2023-01-11T20:50:46.0908541Z dist init r=2, world=4
2023-01-11T20:50:46.0908712Z dist init r=0, world=4
2023-01-11T20:50:46.0908873Z dist init r=1, world=4
2023-01-11T20:50:46.0909074Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T20:50:46.0909306Z   test_non_root (__main__.TestClipGradNorm)
2023-01-11T20:50:46.0909765Z Tests that calling ``clip_grad_norm_()`` on a non-root FSDP instance ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8170
2023-01-11T20:50:46.0910142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8171
2023-01-11T20:50:46.0910463Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8172
2023-01-11T20:50:46.0910780Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8173
2023-01-11T20:50:46.0911262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:46.0911585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:46.0912006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:46.0912343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:46.0912658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:50:46.0913119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:46.0913440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:46.0913847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:46.0914216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:46.0914546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:50:46.0915006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:46.0915313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:46.0915730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:46.0916065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:46.0916392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:50:46.0916863Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:46.0917356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:46.0917680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:46.0918091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:46.0918430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:46.0918758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:50:46.0919244Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:46.0919744Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:46.0920251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:46.0920634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:50:46.0920974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:50:46.0921300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:50:46.0921629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:50:46.0921882Z dist init r=2, world=4
2023-01-11T20:50:46.0922047Z dist init r=3, world=4
2023-01-11T20:50:46.0922219Z dist init r=1, world=4
2023-01-11T20:50:46.0922392Z dist init r=0, world=4
2023-01-11T20:50:46.0922582Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T20:50:46.0922717Z 
2023-01-11T20:50:46.0922918Z ----------------------------------------------------------------------
2023-01-11T20:50:46.0923155Z Ran 3 tests in 5.506s
2023-01-11T20:50:46.0923266Z 
2023-01-11T20:50:46.0923336Z OK (skipped=3)
2023-01-11T20:50:46.0923469Z 
2023-01-11T20:50:46.0923554Z Generating XML reports...
2023-01-11T20:50:46.0924008Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20230111205040.xml
2023-01-11T20:50:46.0924272Z 
2023-01-11T20:50:46.0924571Z ##[endgroup]
2023-01-11T20:50:46.0925025Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_clip_grad_norm_s8_fmiuz)
2023-01-11T20:50:46.0925289Z 
2023-01-11T20:50:46.0925507Z Running distributed/tensor/parallel/test_tp_examples ... [2023-01-11 20:50:46.084097]
2023-01-11T20:50:46.0926045Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_examples.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:50:46.084446]
2023-01-11T20:50:53.2646262Z 
2023-01-11T20:50:53.2647007Z Expand the folded group to see the log file of distributed/tensor/parallel/test_tp_examples
2023-01-11T20:50:53.2648437Z ##[group]PRINTING LOG FILE of distributed/tensor/parallel/test_tp_examples (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_tp_examples_1n52gj7g)
2023-01-11T20:50:53.2649681Z Test results will be stored in test-reports/python-unittest/distributed.tensor.parallel.test_tp_examples
2023-01-11T20:50:53.2650099Z 
2023-01-11T20:50:53.2650214Z Running tests...
2023-01-11T20:50:53.2650770Z ----------------------------------------------------------------------
2023-01-11T20:50:53.2651658Z   test_mlp_megatron_e2e (__main__.DistTensorParallelExampleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8237
2023-01-11T20:50:53.2652357Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8238
2023-01-11T20:50:53.2652834Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8239
2023-01-11T20:50:53.2653350Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8240
2023-01-11T20:50:53.2654215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:53.2654768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:53.2655551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:53.2656176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:53.2656837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:50:53.2657663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:53.2657996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:53.2658420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:53.2658752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:53.2659076Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:50:53.2659533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:53.2659854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:53.2660263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:53.2660602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:53.2660919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:50:53.2661372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:53.2661681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:53.2662241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:53.2662580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:53.2662887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:50:53.2663247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:50:53.2663609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:50:53.2663964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:50:53.2664309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:50:53.2664794Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:53.2665370Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:53.2665881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:53.2666376Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:53.2666662Z ok (1.972s)
2023-01-11T20:50:53.2667000Z   test_self_attn_megatron_e2e (__main__.DistTensorParallelExampleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8293
2023-01-11T20:50:53.2667396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8294
2023-01-11T20:50:53.2667722Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8295
2023-01-11T20:50:53.2668035Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8296
2023-01-11T20:50:53.2668488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:53.2668803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:53.2669222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:53.2669562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:53.2669866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:50:53.2670314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:53.2670635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:53.2671053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:53.2671382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:53.2671698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:50:53.2672148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:53.2672467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:53.2672874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:53.2673238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:53.2673553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:50:53.2674005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:53.2674312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:53.2674776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:53.2675114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:53.2675419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:50:53.2675778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:50:53.2676137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:50:53.2676494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:50:53.2676833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:50:53.2677402Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:53.2677959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:53.2678472Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:53.2678964Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:53.2679280Z skip: Need at least 4 CUDA devices (1.822s)
2023-01-11T20:50:53.2679830Z   test_self_attn_replacement_megatron_e2e (__main__.DistTensorParallelExampleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8349
2023-01-11T20:50:53.2680396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8350
2023-01-11T20:50:53.2680825Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8351
2023-01-11T20:50:53.2681220Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8352
2023-01-11T20:50:53.2681682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:53.2681995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:53.2682417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:53.2682958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:53.2683513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:50:53.2684340Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:53.2684971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:53.2685757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:53.2686367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:53.2686675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:50:53.2687127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:53.2687448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:53.2687854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:53.2688190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:53.2688504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:50:53.2688952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:50:53.2689331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:50:53.2689752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:50:53.2690088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:50:53.2690401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:50:53.2690744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:50:53.2691406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:50:53.2692075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:50:53.2692434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:50:53.2693021Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:53.2693540Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:53.2694051Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:53.2694544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:50:53.2694861Z skip: Need at least 4 CUDA devices (1.822s)
2023-01-11T20:50:53.2694996Z 
2023-01-11T20:50:53.2695194Z ----------------------------------------------------------------------
2023-01-11T20:50:53.2695433Z Ran 3 tests in 5.616s
2023-01-11T20:50:53.2695533Z 
2023-01-11T20:50:53.2695604Z OK (skipped=2)
2023-01-11T20:50:53.2695708Z 
2023-01-11T20:50:53.2695791Z Generating XML reports...
2023-01-11T20:50:53.2696391Z Generated XML report: test-reports/python-unittest/distributed.tensor.parallel.test_tp_examples/TEST-DistTensorParallelExampleTest-20230111205047.xml
2023-01-11T20:50:53.2696714Z 
2023-01-11T20:50:53.2697003Z ##[endgroup]
2023-01-11T20:50:53.2697502Z FINISHED PRINTING LOG FILE of distributed/tensor/parallel/test_tp_examples (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_tp_examples_1n52gj7g)
2023-01-11T20:50:53.2697789Z 
2023-01-11T20:50:53.2697994Z Running distributed/fsdp/test_fsdp_hybrid_shard ... [2023-01-11 20:50:53.265205]
2023-01-11T20:50:53.2698518Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_hybrid_shard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:50:53.265668]
2023-01-11T20:51:01.5419666Z 
2023-01-11T20:51:01.5420374Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_hybrid_shard
2023-01-11T20:51:01.5424462Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_hybrid_shard (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_hybrid_shard_e_tqx2op)
2023-01-11T20:51:01.5425311Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_hybrid_shard
2023-01-11T20:51:01.5425588Z 
2023-01-11T20:51:01.5425674Z Running tests...
2023-01-11T20:51:01.5425969Z ----------------------------------------------------------------------
2023-01-11T20:51:01.5426314Z   test_fsdp_hybrid_shard_basic_setup (__main__.TestFSDPHybridShard)
2023-01-11T20:51:01.5426711Z Tests basic functionality of HYBRID_SHARD and _HYBRID_SHARD_ZERO2: ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8416
2023-01-11T20:51:01.5427160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8417
2023-01-11T20:51:01.5427615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:01.5428008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:01.5428564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:01.5429172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:01.5429547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:01.5430020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:01.5430402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:01.5430815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:01.5431204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:01.5431540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:01.5432156Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:51:01.5432726Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:51:01.5433122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:01.5433506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:01.5441368Z dist init r=0, world=2
2023-01-11T20:51:01.5441593Z dist init r=1, world=2
2023-01-11T20:51:01.5441812Z skip: Need at least 2 CUDA devices (1.860s)
2023-01-11T20:51:01.5442221Z   test_hybrid_shard_pg_mismatch_raises (__main__.TestFSDPHybridShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8444
2023-01-11T20:51:01.5442627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8445
2023-01-11T20:51:01.5443145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:01.5443486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:01.5443925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:01.5444275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:01.5444633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:01.5445186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:01.5445518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:01.5445928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:01.5446268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:01.5446605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:01.5447096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:51:01.5447601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:51:01.5447981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:01.5448320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:01.5448564Z dist init r=1, world=2
2023-01-11T20:51:01.5448739Z dist init r=0, world=2
2023-01-11T20:51:01.5448943Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:51:01.5449304Z   test_invalid_pg_specification_raises (__main__.TestFSDPHybridShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8472
2023-01-11T20:51:01.5449831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8473
2023-01-11T20:51:01.5450287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:01.5450614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:01.5451190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:01.5451536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:01.5451879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:01.5452347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:01.5452659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:01.5453176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:01.5453525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:01.5453866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:01.5454349Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:51:01.5454867Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:51:01.5455254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:01.5455590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:01.5455853Z dist init r=1, world=2
2023-01-11T20:51:01.5456036Z dist init r=0, world=2
2023-01-11T20:51:01.5456244Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:51:01.5456691Z   test_raises_manual_wrap_hybrid_shard_when_none_policy (__main__.TestFSDPHybridShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8500
2023-01-11T20:51:01.5457104Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8501
2023-01-11T20:51:01.5457561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:01.5457879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:01.5458310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:01.5458659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:01.5458993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:01.5459447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:01.5459776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:01.5460388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:01.5460777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:01.5461095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:01.5461592Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:51:01.5462109Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:51:01.5462481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:01.5462832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:01.5463170Z dist init r=1, world=2
2023-01-11T20:51:01.5463350Z dist init r=0, world=2
2023-01-11T20:51:01.5463541Z skip: Need at least 2 CUDA devices (1.612s)
2023-01-11T20:51:01.5463678Z 
2023-01-11T20:51:01.5463881Z ----------------------------------------------------------------------
2023-01-11T20:51:01.5464122Z Ran 4 tests in 6.698s
2023-01-11T20:51:01.5464235Z 
2023-01-11T20:51:01.5464294Z OK (skipped=4)
2023-01-11T20:51:01.5464399Z 
2023-01-11T20:51:01.5464483Z Generating XML reports...
2023-01-11T20:51:01.5464951Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_hybrid_shard/TEST-TestFSDPHybridShard-20230111205054.xml
2023-01-11T20:51:01.5465219Z 
2023-01-11T20:51:01.5465511Z ##[endgroup]
2023-01-11T20:51:01.5465981Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_hybrid_shard (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_hybrid_shard_e_tqx2op)
2023-01-11T20:51:01.5466251Z 
2023-01-11T20:51:01.5466514Z Running distributed/tensor/parallel/test_2d_parallel ... [2023-01-11 20:51:01.542341]
2023-01-11T20:51:01.5467061Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/tensor/parallel/test_2d_parallel.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:51:01.542728]
2023-01-11T20:51:12.1477113Z 
2023-01-11T20:51:12.1477916Z Expand the folded group to see the log file of distributed/tensor/parallel/test_2d_parallel
2023-01-11T20:51:12.1479140Z ##[group]PRINTING LOG FILE of distributed/tensor/parallel/test_2d_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_2d_parallel_nnf1ewp9)
2023-01-11T20:51:12.1480449Z Test results will be stored in test-reports/python-unittest/distributed.tensor.parallel.test_2d_parallel
2023-01-11T20:51:12.1480858Z 
2023-01-11T20:51:12.1480989Z Running tests...
2023-01-11T20:51:12.1481404Z ----------------------------------------------------------------------
2023-01-11T20:51:12.1481844Z   test_2d_fsdp_integration_correctness (__main__.Test2dParallelIntegration) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8539
2023-01-11T20:51:12.1482255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8540
2023-01-11T20:51:12.1482580Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8541
2023-01-11T20:51:12.1482885Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8542
2023-01-11T20:51:12.1483339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1483671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1484331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1484779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1485390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:51:12.1485911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1486223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1486651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1486992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1487312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:51:12.1487756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1488080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1488560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1489230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1489569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:12.1490029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1490353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1490764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1491265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1491581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:12.1491944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:12.1492385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:12.1492886Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1493281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:51:12.1493630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:51:12.1494115Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1494632Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1495139Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1495447Z skip: Need at least 4 CUDA devices (1.966s)
2023-01-11T20:51:12.1495828Z   test_2d_fsdp_integration_fsdp_nested (__main__.Test2dParallelIntegration) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8595
2023-01-11T20:51:12.1496237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8596
2023-01-11T20:51:12.1496622Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8597
2023-01-11T20:51:12.1496928Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8598
2023-01-11T20:51:12.1497369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1497697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1498110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1498453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1498776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:12.1499231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1499543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1499968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1500311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1500629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:51:12.1501067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1501389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1501810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1502207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1502527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:12.1502985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1503306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1503714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1504049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1504368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:51:12.1504714Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:12.1505112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:12.1505476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:51:12.1505833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:51:12.1506308Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1506821Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1507332Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1507836Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1508144Z skip: Need at least 4 CUDA devices (1.919s)
2023-01-11T20:51:12.1508533Z   test_2d_fsdp_integration_fsdp_nested_param_groups (__main__.Test2dParallelIntegration) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8651
2023-01-11T20:51:12.1508950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8652
2023-01-11T20:51:12.1509274Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8653
2023-01-11T20:51:12.1509577Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8654
2023-01-11T20:51:12.1510017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1510345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1510757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1511100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1511424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:12.1511876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1512190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1512609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1513037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1513347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:51:12.1513810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1514134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1514603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1514929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1515246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:51:12.1515696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1516015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1516423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1516762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1517079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:12.1517459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:12.1517826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:51:12.1518206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:12.1518565Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:51:12.1519044Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1519559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1520070Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1520578Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1520886Z skip: Need at least 4 CUDA devices (1.719s)
2023-01-11T20:51:12.1521262Z   test_2d_fsdp_integration_functionality (__main__.Test2dParallelIntegration) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8707
2023-01-11T20:51:12.1521735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8708
2023-01-11T20:51:12.1522188Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8709
2023-01-11T20:51:12.1522648Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8710
2023-01-11T20:51:12.1523275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1523770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1524448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1524989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1525504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:51:12.1526255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1526735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1527445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1528028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1528502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:12.1529193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1529668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1530422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1531033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1531526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:12.1532256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1532760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1533440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1534004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1534546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:51:12.1535250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:12.1535821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:12.1536463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:51:12.1537254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1537861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:51:12.1538666Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1539536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1540331Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1540840Z skip: Need at least 4 CUDA devices (1.720s)
2023-01-11T20:51:12.1541440Z   test_2d_fsdp_integration_use_orig_params (__main__.Test2dParallelIntegration) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8763
2023-01-11T20:51:12.1542108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8764
2023-01-11T20:51:12.1542629Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8765
2023-01-11T20:51:12.1543116Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8766
2023-01-11T20:51:12.1543807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1544333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1545035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1545570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1546045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:12.1546716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1547177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1547840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1548356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1548834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:51:12.1549542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1550174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1550844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1551412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1551939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:12.1552787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:12.1553349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:12.1554124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:12.1554722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:12.1555342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:51:12.1555934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:12.1556553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:12.1557172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:51:12.1557767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:51:12.1558629Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1559751Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1560682Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1561569Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:12.1562127Z skip: Need at least 4 CUDA devices (1.718s)
2023-01-11T20:51:12.1562359Z 
2023-01-11T20:51:12.1562701Z ----------------------------------------------------------------------
2023-01-11T20:51:12.1563122Z Ran 5 tests in 9.042s
2023-01-11T20:51:12.1563308Z 
2023-01-11T20:51:12.1563431Z OK (skipped=5)
2023-01-11T20:51:12.1563603Z 
2023-01-11T20:51:12.1563749Z Generating XML reports...
2023-01-11T20:51:12.1564651Z Generated XML report: test-reports/python-unittest/distributed.tensor.parallel.test_2d_parallel/TEST-Test2dParallelIntegration-20230111205102.xml
2023-01-11T20:51:12.1565198Z 
2023-01-11T20:51:12.1565668Z ##[endgroup]
2023-01-11T20:51:12.1566585Z FINISHED PRINTING LOG FILE of distributed/tensor/parallel/test_2d_parallel (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_2d_parallel_nnf1ewp9)
2023-01-11T20:51:12.1567121Z 
2023-01-11T20:51:12.1567480Z Running distributed/_tensor/test_api ... [2023-01-11 20:51:12.148295]
2023-01-11T20:51:12.1568374Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_api.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:51:12.148759]
2023-01-11T20:51:22.8475982Z 
2023-01-11T20:51:22.8476428Z Expand the folded group to see the log file of distributed/_tensor/test_api
2023-01-11T20:51:22.8477189Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_api_z63js43z)
2023-01-11T20:51:22.8477917Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_api
2023-01-11T20:51:22.8478116Z 
2023-01-11T20:51:22.8478206Z Running tests...
2023-01-11T20:51:22.8478544Z ----------------------------------------------------------------------
2023-01-11T20:51:22.8478939Z   test_distribute_module (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8830
2023-01-11T20:51:22.8479567Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8831
2023-01-11T20:51:22.8479934Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8832
2023-01-11T20:51:22.8480257Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8833
2023-01-11T20:51:22.8480780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8481115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8481647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8481992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8482420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:22.8483026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8483479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8483909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8484252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8484557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:51:22.8485015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8485436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8485955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8486351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8486679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:22.8487194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8487505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8487999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8488342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8488718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:51:22.8489069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:22.8489493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:22.8489860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:51:22.8490276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:51:22.8490754Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8491479Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8492059Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8492611Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8492897Z ok (2.070s)
2023-01-11T20:51:22.8493285Z   test_distribute_module_input_fn_output_fn (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8886
2023-01-11T20:51:22.8493774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8887
2023-01-11T20:51:22.8494138Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8888
2023-01-11T20:51:22.8494454Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8889
2023-01-11T20:51:22.8494965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8495292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8495766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8496112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8496548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:22.8497061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8497423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8497877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8498238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8498545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:51:22.8499069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8499416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8499861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8500247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8500576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:22.8501088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8501400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8501889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8502229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8502602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:51:22.8502954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:22.8503368Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:51:22.8503736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:22.8504132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:51:22.8504633Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8505208Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8505781Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8506280Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8506629Z ok (1.719s)
2023-01-11T20:51:22.8506938Z   test_distribute_tensor (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8942
2023-01-11T20:51:22.8507421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8943
2023-01-11T20:51:22.8507732Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 8944
2023-01-11T20:51:22.8508106Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 8945
2023-01-11T20:51:22.8508556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8508926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8509351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8509751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8510068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:51:22.8510609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8510935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8511420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8511749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8512131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:22.8512590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8512970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8513380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8513722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8514046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:51:22.8514558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8514869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8515293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8515648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8515952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:22.8516312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:22.8516734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:22.8517096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:51:22.8517571Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8517965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:51:22.8518447Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8518955Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8519453Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8519734Z ok (1.719s)
2023-01-11T20:51:22.8520051Z   test_distribute_tensor_errors (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8998
2023-01-11T20:51:22.8520519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8999
2023-01-11T20:51:22.8520843Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9000
2023-01-11T20:51:22.8521156Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9001
2023-01-11T20:51:22.8521601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8521914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8522334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8522674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8522979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:22.8523465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8523792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8524218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8524544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8524863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:51:22.8525317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8525636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8526041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8526379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8526701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:51:22.8527181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8527515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8527938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8528277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8528581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:22.8528944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:22.8529312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:22.8529662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:51:22.8530017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:51:22.8530503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8531119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8531617Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8532124Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8532512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T20:51:22.8532872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T20:51:22.8533284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T20:51:22.8533633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T20:51:22.8534114Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:51:22.8534621Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:51:22.8534994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T20:51:22.8535353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T20:51:22.8535876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:51:22.8536438Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:51:22.8536818Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T20:51:22.8537174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T20:51:22.8537658Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:51:22.8538150Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:51:22.8538536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T20:51:22.8539017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:51:22.8539525Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:51:22.8539900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T20:51:22.8540258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T20:51:22.8540684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T20:51:22.8541168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:51:22.8541659Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:51:22.8542048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T20:51:22.8542411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T20:51:22.8542891Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:51:22.8543382Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:51:22.8543765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T20:51:22.8544121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T20:51:22.8544604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:51:22.8545093Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:51:22.8545601Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:51:22.8546152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:51:22.8546420Z ok (1.920s)
2023-01-11T20:51:22.8546750Z   test_distribute_tensor_uneven_sharding (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9078
2023-01-11T20:51:22.8547141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9079
2023-01-11T20:51:22.8547464Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9080
2023-01-11T20:51:22.8547768Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9081
2023-01-11T20:51:22.8548211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8548539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8549004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8549337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8549657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:22.8550112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8550423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8550844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8551183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8551506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:22.8551952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8552276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8552702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8553027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8553346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:51:22.8553796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:22.8554117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:22.8554524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:22.8554859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:22.8555180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:51:22.8555536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:22.8555883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:22.8556244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:51:22.8556598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:51:22.8557073Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8557585Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8558099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8558641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:22.8558908Z ok (1.719s)
2023-01-11T20:51:22.8559011Z 
2023-01-11T20:51:22.8559209Z ----------------------------------------------------------------------
2023-01-11T20:51:22.8559450Z Ran 5 tests in 9.148s
2023-01-11T20:51:22.8559563Z 
2023-01-11T20:51:22.8559623Z OK
2023-01-11T20:51:22.8559701Z 
2023-01-11T20:51:22.8559784Z Generating XML reports...
2023-01-11T20:51:22.8560210Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_api/TEST-DTensorAPITest-20230111205113.xml
2023-01-11T20:51:22.8560451Z 
2023-01-11T20:51:22.8560747Z ##[endgroup]
2023-01-11T20:51:22.8561158Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_api_z63js43z)
2023-01-11T20:51:22.8561402Z 
2023-01-11T20:51:22.8561685Z Running distributed/elastic/timer/local_timer_example ... [2023-01-11 20:51:22.848084]
2023-01-11T20:51:22.8562231Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:51:22.848477]
2023-01-11T20:51:34.1661495Z 
2023-01-11T20:51:34.1662389Z Expand the folded group to see the log file of distributed/elastic/timer/local_timer_example
2023-01-11T20:51:34.1663581Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/local_timer_example (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_example_n9ucyi8c)
2023-01-11T20:51:34.1664524Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_example
2023-01-11T20:51:34.1664754Z 
2023-01-11T20:51:34.1664896Z Running tests...
2023-01-11T20:51:34.1665202Z ----------------------------------------------------------------------
2023-01-11T20:51:34.1665741Z   test_example_start_method_spawn (__main__.LocalTimerExample) ... [INFO] 2023-01-11 20:51:24,306 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2023-01-11T20:51:34.1666207Z [INFO] 2023-01-11 20:51:24,306 api: Starting watchdog thread...
2023-01-11T20:51:34.1666680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1667127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1667602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1667938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1668356Z [INFO] 2023-01-11 20:51:26,374 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1668837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1669159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1669655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1670080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1670584Z [INFO] 2023-01-11 20:51:26,375 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1671000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1671321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1671740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1672080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1672426Z [INFO] 2023-01-11 20:51:26,375 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1673077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1673406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1673815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1674158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1674522Z [INFO] 2023-01-11 20:51:26,394 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1674958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1675270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1675694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1676101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1676452Z [INFO] 2023-01-11 20:51:26,432 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1676878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1677200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1677626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1677956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1678315Z [INFO] 2023-01-11 20:51:26,433 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1678742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1679051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1679477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1679820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1680176Z [INFO] 2023-01-11 20:51:26,433 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1680585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1680908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1681329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1681671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1682016Z [INFO] 2023-01-11 20:51:26,479 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1682448Z [INFO] 2023-01-11 20:51:27,519 api: Reaping worker_id=[9148]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2023-01-11T20:51:34.1682826Z [INFO] 2023-01-11 20:51:27,519 api: Successfully reaped worker=[9148]
2023-01-11T20:51:34.1683227Z [INFO] 2023-01-11 20:51:27,530 api: Reaping worker_id=[9150]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2023-01-11T20:51:34.1683595Z [INFO] 2023-01-11 20:51:27,530 api: Successfully reaped worker=[9150]
2023-01-11T20:51:34.1684004Z [INFO] 2023-01-11 20:51:27,550 api: Reaping worker_id=[9146]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2023-01-11T20:51:34.1684373Z [INFO] 2023-01-11 20:51:27,550 api: Successfully reaped worker=[9146]
2023-01-11T20:51:34.1684766Z [INFO] 2023-01-11 20:51:27,550 api: Reaping worker_id=[9152]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2023-01-11T20:51:34.1685129Z [INFO] 2023-01-11 20:51:27,550 api: Successfully reaped worker=[9152]
2023-01-11T20:51:34.1685447Z [INFO] 2023-01-11 20:51:27,559 api: Stopping LocalTimerServer
2023-01-11T20:51:34.1685785Z [INFO] 2023-01-11 20:51:27,559 api: Stopping watchdog thread...
2023-01-11T20:51:34.1685993Z ok (3.509s)
2023-01-11T20:51:34.1686412Z   test_torch_mp_example (__main__.LocalTimerExample) ... [INFO] 2023-01-11 20:51:27,563 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2023-01-11T20:51:34.1686806Z [INFO] 2023-01-11 20:51:27,563 api: Starting watchdog thread...
2023-01-11T20:51:34.1687204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1687536Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1687956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1688284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1688645Z [INFO] 2023-01-11 20:51:29,551 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1689110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1689438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1689847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1690185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1690546Z [INFO] 2023-01-11 20:51:29,593 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1691144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1691458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1691884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1692227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1692575Z [INFO] 2023-01-11 20:51:29,604 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1693007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1693333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1693753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1694078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1694436Z [INFO] 2023-01-11 20:51:29,606 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1694862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1695172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1695596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1695941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1696301Z [INFO] 2023-01-11 20:51:29,614 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1696791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1697115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1697542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1697870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1698232Z [INFO] 2023-01-11 20:51:29,628 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1698664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1699076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1699489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1699833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1700194Z [INFO] 2023-01-11 20:51:29,630 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1700619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1700929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1701348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1701689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1702074Z [INFO] 2023-01-11 20:51:29,635 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1702507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1702828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1703247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1703571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1703926Z [INFO] 2023-01-11 20:51:32,642 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1704349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1704653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1705074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1705417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1705777Z [INFO] 2023-01-11 20:51:32,660 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1706192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1706513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1706933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1707261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1707617Z [INFO] 2023-01-11 20:51:32,674 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1708041Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1708369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1708777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1709116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1709469Z [INFO] 2023-01-11 20:51:32,674 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1709891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1710197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1710614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1710953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1711295Z [INFO] 2023-01-11 20:51:32,678 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1711763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1712082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1712499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1712823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1713177Z [INFO] 2023-01-11 20:51:32,688 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1713605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1713913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1714332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1714708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1715071Z [INFO] 2023-01-11 20:51:32,706 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1715483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:34.1715804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:34.1716224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:34.1716563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:34.1716906Z [INFO] 2023-01-11 20:51:32,718 api: Timer client configured to: LocalTimerClient
2023-01-11T20:51:34.1717328Z [INFO] 2023-01-11 20:51:33,783 api: Reaping worker_id=[9328]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2023-01-11T20:51:34.1717709Z [INFO] 2023-01-11 20:51:33,783 api: Successfully reaped worker=[9328]
2023-01-11T20:51:34.1718110Z [INFO] 2023-01-11 20:51:33,804 api: Reaping worker_id=[9326]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2023-01-11T20:51:34.1718479Z [INFO] 2023-01-11 20:51:33,804 api: Successfully reaped worker=[9326]
2023-01-11T20:51:34.1718885Z [INFO] 2023-01-11 20:51:33,804 api: Reaping worker_id=[9324]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2023-01-11T20:51:34.1719288Z [INFO] 2023-01-11 20:51:33,804 local_timer: Process with pid=9324 does not exist. Skipping
2023-01-11T20:51:34.1719622Z [INFO] 2023-01-11 20:51:33,804 api: Successfully reaped worker=[9324]
2023-01-11T20:51:34.1720035Z [INFO] 2023-01-11 20:51:33,814 api: Reaping worker_id=[9323]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2023-01-11T20:51:34.1720439Z [INFO] 2023-01-11 20:51:33,814 local_timer: Process with pid=9323 does not exist. Skipping
2023-01-11T20:51:34.1720770Z [INFO] 2023-01-11 20:51:33,814 api: Successfully reaped worker=[9323]
2023-01-11T20:51:34.1721180Z [INFO] 2023-01-11 20:51:33,814 api: Reaping worker_id=[9325]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2023-01-11T20:51:34.1721581Z [INFO] 2023-01-11 20:51:33,814 local_timer: Process with pid=9325 does not exist. Skipping
2023-01-11T20:51:34.1721924Z [INFO] 2023-01-11 20:51:33,814 api: Successfully reaped worker=[9325]
2023-01-11T20:51:34.1722322Z [INFO] 2023-01-11 20:51:33,814 api: Reaping worker_id=[9329]. Expired timers: ['/opt/conda/lib/python3.7/contextlib.py#112']
2023-01-11T20:51:34.1722693Z [INFO] 2023-01-11 20:51:33,814 api: Successfully reaped worker=[9329]
2023-01-11T20:51:34.1723010Z [INFO] 2023-01-11 20:51:33,828 api: Stopping LocalTimerServer
2023-01-11T20:51:34.1723307Z [INFO] 2023-01-11 20:51:33,828 api: Stopping watchdog thread...
2023-01-11T20:51:34.1723513Z ok (6.274s)
2023-01-11T20:51:34.1723619Z 
2023-01-11T20:51:34.1723818Z ----------------------------------------------------------------------
2023-01-11T20:51:34.1724059Z Ran 2 tests in 9.783s
2023-01-11T20:51:34.1724159Z 
2023-01-11T20:51:34.1724260Z OK
2023-01-11T20:51:34.1724352Z 
2023-01-11T20:51:34.1724436Z Generating XML reports...
2023-01-11T20:51:34.1724912Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20230111205124.xml
2023-01-11T20:51:34.1725193Z 
2023-01-11T20:51:34.1725464Z ##[endgroup]
2023-01-11T20:51:34.1725963Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/local_timer_example (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_example_n9ucyi8c)
2023-01-11T20:51:34.1726251Z 
2023-01-11T20:51:34.1726462Z Running distributed/_tensor/test_matrix_ops ... [2023-01-11 20:51:34.166559]
2023-01-11T20:51:34.1726980Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_matrix_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:51:34.166920]
2023-01-11T20:51:48.5343481Z 
2023-01-11T20:51:48.5344191Z Expand the folded group to see the log file of distributed/_tensor/test_matrix_ops
2023-01-11T20:51:48.5345508Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_matrix_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_matrix_ops_iox9l0ec)
2023-01-11T20:51:48.5346395Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_matrix_ops
2023-01-11T20:51:48.5346735Z 
2023-01-11T20:51:48.5346847Z Running tests...
2023-01-11T20:51:48.5347332Z ----------------------------------------------------------------------
2023-01-11T20:51:48.5347915Z   test_addmm (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9422
2023-01-11T20:51:48.5350080Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9423
2023-01-11T20:51:48.5350589Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9424
2023-01-11T20:51:48.5351122Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9425
2023-01-11T20:51:48.5351988Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5352566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5353398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5354094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5354604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:51:48.5355383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5355984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5356785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5357425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5361291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:51:48.5362219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5362840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5363668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5364249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5364824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:48.5365626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5366132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5366823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5372285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5372970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:48.5373557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:48.5374195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:48.5374847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:51:48.5375508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:51:48.5376533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5377727Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5378293Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5378815Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5379096Z ok (1.966s)
2023-01-11T20:51:48.5379417Z   test_addmm_auto_redistribute (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9478
2023-01-11T20:51:48.5379790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9479
2023-01-11T20:51:48.5380277Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9480
2023-01-11T20:51:48.5380789Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9481
2023-01-11T20:51:48.5381614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5381939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5382368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5382711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5383019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:51:48.5383476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5383797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5384220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5384551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5384873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:51:48.5401655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5402368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5403126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5403655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5404149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:48.5404890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5405483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5406274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5407018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5407545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:48.5408177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:48.5408871Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:51:48.5409536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:48.5410445Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5411281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:51:48.5411859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5412363Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5413028Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5413565Z ok (1.720s)
2023-01-11T20:51:48.5413947Z   test_baddbmm (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9534
2023-01-11T20:51:48.5414308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9535
2023-01-11T20:51:48.5414631Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9536
2023-01-11T20:51:48.5414954Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9537
2023-01-11T20:51:48.5415445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5416005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5416674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5417218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5417554Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:51:48.5418015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5418548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5419241Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5419808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5420408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:48.5421260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5421833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5422517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5422858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5423175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:51:48.5423619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5423943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5424369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5424801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5425107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:48.5425466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:48.5425827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:48.5426172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:51:48.5426658Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5427050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:51:48.5427580Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5428085Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5428593Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5428907Z skip: Need at least 4 CUDA devices (1.721s)
2023-01-11T20:51:48.5429233Z   test_bmm (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9590
2023-01-11T20:51:48.5429581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9591
2023-01-11T20:51:48.5429905Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9592
2023-01-11T20:51:48.5430223Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9593
2023-01-11T20:51:48.5430656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5430983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5431408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5431864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5432172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:48.5432625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5432948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5433368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5433696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5434017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:48.5434468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5434781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5435203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5435541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5435857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:51:48.5436295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5436616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5437040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5437408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5437730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:51:48.5438087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:48.5438447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:51:48.5438919Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5439313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:48.5439669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:51:48.5440211Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5440707Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5441217Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5441501Z ok (2.120s)
2023-01-11T20:51:48.5441781Z   test_mm (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9658
2023-01-11T20:51:48.5442144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9659
2023-01-11T20:51:48.5442469Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9660
2023-01-11T20:51:48.5442788Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9661
2023-01-11T20:51:48.5443222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5443552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5443977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5444320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5444626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:48.5445080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5445400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5445812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5446149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5446469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:48.5446919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5447228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5447648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5447987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5448292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:51:48.5448744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5449067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5449489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5449855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5450176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:51:48.5450535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:48.5451067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:48.5451422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:51:48.5451780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:51:48.5452273Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5452841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5453364Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5453961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5454246Z ok (1.820s)
2023-01-11T20:51:48.5454528Z   test_t (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9714
2023-01-11T20:51:48.5454890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9715
2023-01-11T20:51:48.5455212Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9716
2023-01-11T20:51:48.5455531Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9717
2023-01-11T20:51:48.5455963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5456290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5456781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5457111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5457432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:48.5457882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5458253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5458664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5459000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5459322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:51:48.5459814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5460123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5460545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5460884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5461201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:51:48.5461640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5461963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5462385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5462780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5463085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:48.5463443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:48.5463808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:48.5464284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5464674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:51:48.5465032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:51:48.5465552Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5466054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5466560Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5466846Z ok (1.719s)
2023-01-11T20:51:48.5467149Z   test_t_partial (__main__.DistMatrixOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9770
2023-01-11T20:51:48.5467506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9771
2023-01-11T20:51:48.5467827Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9772
2023-01-11T20:51:48.5468143Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9773
2023-01-11T20:51:48.5468578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5468906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5469330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5469667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5469971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:51:48.5470424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5470746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5471157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5471494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5471815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:51:48.5472272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5472580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5473001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5473342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5473660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:51:48.5474100Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:51:48.5474422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:51:48.5474848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:51:48.5475211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:51:48.5475530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:51:48.5475884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:51:48.5476249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:51:48.5476720Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5477110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:51:48.5477465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:51:48.5477989Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5478486Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5478996Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:51:48.5479281Z ok (1.719s)
2023-01-11T20:51:48.5479382Z 
2023-01-11T20:51:48.5479568Z ----------------------------------------------------------------------
2023-01-11T20:51:48.5479809Z Ran 7 tests in 12.786s
2023-01-11T20:51:48.5479920Z 
2023-01-11T20:51:48.5479991Z OK (skipped=1)
2023-01-11T20:51:48.5480095Z 
2023-01-11T20:51:48.5480178Z Generating XML reports...
2023-01-11T20:51:48.5480607Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_matrix_ops/TEST-DistMatrixOpsTest-20230111205135.xml
2023-01-11T20:51:48.5480861Z 
2023-01-11T20:51:48.5481245Z ##[endgroup]
2023-01-11T20:51:48.5481692Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_matrix_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_matrix_ops_iox9l0ec)
2023-01-11T20:51:48.5481945Z 
2023-01-11T20:51:48.5482124Z Running distributed/fsdp/test_fsdp_meta ... [2023-01-11 20:51:48.534785]
2023-01-11T20:51:48.5482628Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:51:48.535159]
2023-01-11T20:52:03.2804186Z 
2023-01-11T20:52:03.2805028Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_meta
2023-01-11T20:52:03.2810127Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_meta_k82nhmgz)
2023-01-11T20:52:03.2811374Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_meta
2023-01-11T20:52:03.2811695Z 
2023-01-11T20:52:03.2811876Z Running tests...
2023-01-11T20:52:03.2812447Z ----------------------------------------------------------------------
2023-01-11T20:52:03.2813107Z   test_bad_arg_meta (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9837
2023-01-11T20:52:03.2813791Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9838
2023-01-11T20:52:03.2814670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:03.2815279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:03.2816054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:03.2816807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:03.2817426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:03.2819405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:03.2820288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:03.2821090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:03.2821729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:03.2822327Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:03.2823217Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:03.2824118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:03.2824834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:03.2825575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:03.2826041Z dist init r=1, world=2
2023-01-11T20:52:03.2826372Z dist init r=0, world=2
2023-01-11T20:52:03.2826734Z skip: Need at least 2 CUDA devices (1.863s)
2023-01-11T20:52:03.2827417Z   test_bad_arg_torchdistx (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2023-01-11T20:52:03.2828089Z   test_meta_device_with_mixed_precision (__main__.TestFSDPWithMetaDevice)
2023-01-11T20:52:03.2828485Z Tests meta device initialization with a ``param_init_fn`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9865
2023-01-11T20:52:03.2828859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9866
2023-01-11T20:52:03.2829314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:03.2829642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:03.2830061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:03.2830406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:03.2830739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:03.2831235Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:03.2831718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:03.2832042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:03.2832467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:03.2832807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:03.2833128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:03.2833623Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:03.2834007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:03.2834336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:03.2834594Z dist init r=0, world=2
2023-01-11T20:52:03.2834772Z dist init r=1, world=2
2023-01-11T20:52:03.2834965Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:52:03.2835359Z   test_nested_model_with_meta_device_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9893
2023-01-11T20:52:03.2835783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9894
2023-01-11T20:52:03.2836285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:03.2836598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:03.2837020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:03.2837358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:03.2837693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:03.2838142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:03.2838464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:03.2838888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:03.2839254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:03.2839585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:03.2840178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:03.2840800Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:03.2841507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:03.2841892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:03.2842146Z dist init r=0, world=2
2023-01-11T20:52:03.2842332Z dist init r=1, world=2
2023-01-11T20:52:03.2842527Z skip: Need at least 2 CUDA devices (1.612s)
2023-01-11T20:52:03.2842930Z   test_nested_model_with_meta_device_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9921
2023-01-11T20:52:03.2843362Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9922
2023-01-11T20:52:03.2843801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:03.2844132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:03.2844560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:03.2844905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:03.2845230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:03.2845694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:03.2846027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:03.2846443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:03.2846786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:03.2847119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:03.2847612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:03.2848119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:03.2848507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:03.2848852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:03.2849113Z dist init r=1, world=2
2023-01-11T20:52:03.2849335Z dist init r=0, world=2
2023-01-11T20:52:03.2849544Z skip: Need at least 2 CUDA devices (1.612s)
2023-01-11T20:52:03.2849942Z   test_nested_model_with_meta_device_reset_params_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9949
2023-01-11T20:52:03.2850358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9950
2023-01-11T20:52:03.2850811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:03.2851293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:03.2851728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:03.2852061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:03.2852467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:03.2852938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:03.2853253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:03.2853680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:03.2854020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:03.2854355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:03.2855089Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:03.2855885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:03.2856560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:03.2857090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:03.2857448Z dist init r=0, world=2
2023-01-11T20:52:03.2857727Z dist init r=1, world=2
2023-01-11T20:52:03.2858089Z skip: Need at least 2 CUDA devices (1.612s)
2023-01-11T20:52:03.2858727Z   test_nested_model_with_meta_device_reset_params_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9977
2023-01-11T20:52:03.2859436Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9978
2023-01-11T20:52:03.2860092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:03.2860421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:03.2860848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:03.2861187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:03.2861517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:03.2861976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:03.2862286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:03.2862706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:03.2863043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:03.2863374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:03.2863851Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:03.2864489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:03.2864870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:03.2865210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:03.2865455Z dist init r=0, world=2
2023-01-11T20:52:03.2865633Z dist init r=1, world=2
2023-01-11T20:52:03.2865837Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:52:03.2866231Z   test_nested_model_with_torchdistX_default_init_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2023-01-11T20:52:03.2866772Z   test_nested_model_with_torchdistX_default_init_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2023-01-11T20:52:03.2867343Z   test_nested_model_with_torchdistX_init_fn_auto_wrap_False (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2023-01-11T20:52:03.2867863Z   test_nested_model_with_torchdistX_init_fn_auto_wrap_True (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2023-01-11T20:52:03.2868366Z   test_simple_model_with_meta_device_default_init (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10005
2023-01-11T20:52:03.2868779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10006
2023-01-11T20:52:03.2869231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:03.2869555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:03.2869969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:03.2870316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:03.2870649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:03.2871116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:03.2871424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:03.2871845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:03.2872179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:03.2872495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:03.2872989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:03.2873503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:03.2873884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:03.2874212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:03.2874460Z dist init r=1, world=2
2023-01-11T20:52:03.2874639Z dist init r=0, world=2
2023-01-11T20:52:03.2874831Z skip: Need at least 2 CUDA devices (1.612s)
2023-01-11T20:52:03.2875209Z   test_simple_model_with_meta_device_reset_params (__main__.TestFSDPWithMetaDevice) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10033
2023-01-11T20:52:03.2875623Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10034
2023-01-11T20:52:03.2876076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:03.2876425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:03.2876848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:03.2877192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:03.2877523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:03.2877977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:03.2878297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:03.2878717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:03.2879042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:03.2879410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:03.2879905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:03.2880421Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:03.2880788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:03.2881131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:03.2881388Z dist init r=1, world=2
2023-01-11T20:52:03.2881553Z dist init r=0, world=2
2023-01-11T20:52:03.2881757Z skip: Need at least 2 CUDA devices (1.612s)
2023-01-11T20:52:03.2882142Z   test_simple_model_with_torchdistX_default_init (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.001s)
2023-01-11T20:52:03.2882649Z   test_simple_model_with_torchdistX_init_fn (__main__.TestFSDPWithMetaDevice) ... skip: Test requires torchdistX: https://github.com/pytorch/torchdistX (0.000s)
2023-01-11T20:52:03.2882909Z 
2023-01-11T20:52:03.2883097Z ----------------------------------------------------------------------
2023-01-11T20:52:03.2883336Z Ran 15 tests in 13.154s
2023-01-11T20:52:03.2883452Z 
2023-01-11T20:52:03.2883525Z OK (skipped=15)
2023-01-11T20:52:03.2883630Z 
2023-01-11T20:52:03.2883713Z Generating XML reports...
2023-01-11T20:52:03.2884161Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20230111205149.xml
2023-01-11T20:52:03.2884430Z 
2023-01-11T20:52:03.2884740Z ##[endgroup]
2023-01-11T20:52:03.2885176Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_meta (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_meta_k82nhmgz)
2023-01-11T20:52:03.2885415Z 
2023-01-11T20:52:03.2885606Z Running distributed/fsdp/test_fsdp_comm ... [2023-01-11 20:52:03.280884]
2023-01-11T20:52:03.2886110Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:52:03.281365]
2023-01-11T20:52:18.8613014Z 
2023-01-11T20:52:18.8613566Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_comm
2023-01-11T20:52:18.8614574Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_t744u3bx)
2023-01-11T20:52:18.8615995Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm
2023-01-11T20:52:18.8616366Z 
2023-01-11T20:52:18.8616772Z Running tests...
2023-01-11T20:52:18.8630148Z ----------------------------------------------------------------------
2023-01-11T20:52:18.8630774Z   test_communication_nested_model_False_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication)
2023-01-11T20:52:18.8631373Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10072
2023-01-11T20:52:18.8632138Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10073
2023-01-11T20:52:18.8632690Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10074
2023-01-11T20:52:18.8633022Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10075
2023-01-11T20:52:18.8633492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8633812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8634247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8634592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8635025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:18.8635492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8635820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8636247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8636574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8636914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:18.8637410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8637908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8638225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8638654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8638998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8639318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:52:18.8639786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8640115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8640543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8640871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8641203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:52:18.8641699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8642217Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8642728Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8643114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:18.8643461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:52:18.8643797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:18.8644129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:52:18.8644380Z dist init r=1, world=4
2023-01-11T20:52:18.8644613Z dist init r=2, world=4
2023-01-11T20:52:18.8644780Z dist init r=0, world=4
2023-01-11T20:52:18.8644956Z dist init r=3, world=4
2023-01-11T20:52:18.8645160Z skip: Need at least 2 CUDA devices (1.966s)
2023-01-11T20:52:18.8645494Z   test_communication_nested_model_False_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2023-01-11T20:52:18.8646064Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10128
2023-01-11T20:52:18.8646456Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10129
2023-01-11T20:52:18.8646787Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10130
2023-01-11T20:52:18.8647099Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10131
2023-01-11T20:52:18.8647545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8647911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8648332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8648677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8649013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:18.8649487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8649800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8650231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8650578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8651065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:18.8651536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8651856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8652280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8652612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8653139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:52:18.8653930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8654701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8655169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8655767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8656245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8656803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:52:18.8657607Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8658440Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8659283Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8659885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:18.8660677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:52:18.8661227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:18.8661728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:52:18.8662087Z dist init r=1, world=4
2023-01-11T20:52:18.8662351Z dist init r=0, world=4
2023-01-11T20:52:18.8662605Z dist init r=3, world=4
2023-01-11T20:52:18.8662844Z dist init r=2, world=4
2023-01-11T20:52:18.8663142Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T20:52:18.8663595Z   test_communication_nested_model_False_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication)
2023-01-11T20:52:18.8664417Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10184
2023-01-11T20:52:18.8665152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10185
2023-01-11T20:52:18.8665697Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10186
2023-01-11T20:52:18.8666223Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10187
2023-01-11T20:52:18.8666994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8667513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8668183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8668730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8669234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:18.8669994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8670501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8671182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8671735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8672267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:18.8672992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8673483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8674174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8674727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8675257Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:52:18.8676028Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8676791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8677306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8678012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8678512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8678978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:52:18.8679697Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8680407Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8681253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8681833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:18.8682346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:52:18.8682901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:52:18.8683434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:18.8683814Z dist init r=3, world=4
2023-01-11T20:52:18.8684082Z dist init r=0, world=4
2023-01-11T20:52:18.8684378Z dist init r=2, world=4
2023-01-11T20:52:18.8684662Z dist init r=1, world=4
2023-01-11T20:52:18.8684967Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T20:52:18.8685696Z   test_communication_nested_model_False_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2023-01-11T20:52:18.8686714Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10240
2023-01-11T20:52:18.8687428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10241
2023-01-11T20:52:18.8687993Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10242
2023-01-11T20:52:18.8688520Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10243
2023-01-11T20:52:18.8689287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8689821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8690566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8691339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8691898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:18.8692710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8693310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8694068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8694618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8695216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:52:18.8696039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8696681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8697452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8698051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8698641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:18.8699484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8700060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8700823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8701438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8702024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:52:18.8703112Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8704046Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8704920Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8705833Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8706515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:52:18.8707136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:18.8707753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:18.8708476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:52:18.8708923Z dist init r=0, world=4
2023-01-11T20:52:18.8709241Z dist init r=3, world=4
2023-01-11T20:52:18.8709538Z dist init r=1, world=4
2023-01-11T20:52:18.8709834Z dist init r=2, world=4
2023-01-11T20:52:18.8710199Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T20:52:18.8710744Z   test_communication_nested_model_True_use_no_sync_False_sharding_strategy_None (__main__.TestCommunication)
2023-01-11T20:52:18.8711709Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10296
2023-01-11T20:52:18.8712424Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10297
2023-01-11T20:52:18.8713013Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10298
2023-01-11T20:52:18.8713596Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10299
2023-01-11T20:52:18.8714426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8714995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8715749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8716340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8716926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:18.8717775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8718348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8719107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8719729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8720328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:18.8721178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8721774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8722545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8723134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8723725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:52:18.8724574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8725159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8726038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8726667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8727221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:52:18.8728116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8729042Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8729951Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8730991Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8731737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:18.8732325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:52:18.8732943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:18.8733544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:52:18.8733978Z dist init r=1, world=4
2023-01-11T20:52:18.8734272Z dist init r=2, world=4
2023-01-11T20:52:18.8734575Z dist init r=0, world=4
2023-01-11T20:52:18.8734848Z dist init r=3, world=4
2023-01-11T20:52:18.8735196Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T20:52:18.8735790Z   test_communication_nested_model_True_use_no_sync_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2023-01-11T20:52:18.8736898Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10352
2023-01-11T20:52:18.8737610Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10353
2023-01-11T20:52:18.8738200Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10354
2023-01-11T20:52:18.8738792Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10355
2023-01-11T20:52:18.8739595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8740180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8740978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8741587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8742158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:18.8742994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8743583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8744343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8744962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8745547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:18.8746384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8746948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8747724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8748497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8749067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:52:18.8750048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8750980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8751882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8752452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8753245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8753844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8754543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:52:18.8755445Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8756381Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8757060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:52:18.8757676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:52:18.8758282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:18.8758889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:18.8759345Z dist init r=2, world=4
2023-01-11T20:52:18.8759626Z dist init r=1, world=4
2023-01-11T20:52:18.8759940Z dist init r=3, world=4
2023-01-11T20:52:18.8760241Z dist init r=0, world=4
2023-01-11T20:52:18.8760580Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T20:52:18.8761129Z   test_communication_nested_model_True_use_no_sync_True_sharding_strategy_None (__main__.TestCommunication)
2023-01-11T20:52:18.8762096Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10408
2023-01-11T20:52:18.8762772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10409
2023-01-11T20:52:18.8763343Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10410
2023-01-11T20:52:18.8763913Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10411
2023-01-11T20:52:18.8764747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8765323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8766096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8766706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8767310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:18.8768158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8768749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8769503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8770120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8770699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:18.8771833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8772392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8773155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8773747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8774312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:52:18.8775319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8775880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8776710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8777462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8778044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:52:18.8778938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8779851Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8780797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8781732Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8782439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:18.8783053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:18.8783698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:52:18.8784310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:52:18.8784781Z dist init r=1, world=4
2023-01-11T20:52:18.8785102Z dist init r=0, world=4
2023-01-11T20:52:18.8785406Z dist init r=3, world=4
2023-01-11T20:52:18.8785722Z dist init r=2, world=4
2023-01-11T20:52:18.8786085Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T20:52:18.8786674Z   test_communication_nested_model_True_use_no_sync_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunication)
2023-01-11T20:52:18.8787660Z Tests FSDP's communication cost in terms of calls to collective ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10464
2023-01-11T20:52:18.8788352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10465
2023-01-11T20:52:18.8788940Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10466
2023-01-11T20:52:18.8789541Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10467
2023-01-11T20:52:18.8790369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8790946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8791737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8792343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8792928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:18.8793757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8794313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8795224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8795833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8796404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:18.8797213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8797805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8798553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8799117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8799630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:52:18.8800602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:18.8801149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:18.8801867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:18.8802477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:18.8803060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:52:18.8803890Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8804774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8805708Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8806639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:18.8807264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:18.8807850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:52:18.8808456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:18.8809065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:52:18.8809489Z dist init r=2, world=4
2023-01-11T20:52:18.8809802Z dist init r=0, world=4
2023-01-11T20:52:18.8810107Z dist init r=1, world=4
2023-01-11T20:52:18.8810383Z dist init r=3, world=4
2023-01-11T20:52:18.8810740Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T20:52:18.8811352Z 
2023-01-11T20:52:18.8811739Z ----------------------------------------------------------------------
2023-01-11T20:52:18.8812154Z Ran 8 tests in 14.008s
2023-01-11T20:52:18.8812334Z 
2023-01-11T20:52:18.8812461Z OK (skipped=8)
2023-01-11T20:52:18.8812659Z 
2023-01-11T20:52:18.8812794Z Generating XML reports...
2023-01-11T20:52:18.8813580Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20230111205204.xml
2023-01-11T20:52:18.8814014Z 
2023-01-11T20:52:18.8814490Z ##[endgroup]
2023-01-11T20:52:18.8815281Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_t744u3bx)
2023-01-11T20:52:18.8815735Z 
2023-01-11T20:52:18.8816126Z Running distributed/fsdp/test_fsdp_freezing_weights ... [2023-01-11 20:52:18.862621]
2023-01-11T20:52:18.8817171Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_freezing_weights.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:52:18.863089]
2023-01-11T20:52:34.5528866Z 
2023-01-11T20:52:34.5543442Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_freezing_weights
2023-01-11T20:52:34.5544599Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_freezing_weights (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_freezing_weights_i3b12so4)
2023-01-11T20:52:34.5545480Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights
2023-01-11T20:52:34.5545775Z 
2023-01-11T20:52:34.5545879Z Running tests...
2023-01-11T20:52:34.5546562Z ----------------------------------------------------------------------
2023-01-11T20:52:34.5547308Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10531
2023-01-11T20:52:34.5548067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10532
2023-01-11T20:52:34.5548857Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10533
2023-01-11T20:52:34.5549362Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10534
2023-01-11T20:52:34.5550059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5550517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5551129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5551597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5552043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:34.5552705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5553160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5553759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5554272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5554757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:34.5555486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5555956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5556596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5557093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5557589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:52:34.5558337Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5559099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5559582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5560208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5560706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5561184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:52:34.5561927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5562685Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5563640Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5564278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:52:34.5564800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:34.5565289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:34.5565794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:52:34.5566184Z dist init r=3, world=4
2023-01-11T20:52:34.5566428Z dist init r=0, world=4
2023-01-11T20:52:34.5566677Z dist init r=1, world=4
2023-01-11T20:52:34.5566920Z dist init r=2, world=4
2023-01-11T20:52:34.5567190Z skip: Need at least 2 CUDA devices (1.969s)
2023-01-11T20:52:34.5567895Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10587
2023-01-11T20:52:34.5568570Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10588
2023-01-11T20:52:34.5569046Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10589
2023-01-11T20:52:34.5569492Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10590
2023-01-11T20:52:34.5570156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5570619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5571404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5571988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5572535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:34.5573058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5573371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5573795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5574136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5574467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:52:34.5574918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5575239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5575666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5575992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5576322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:34.5576870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5577193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5577599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5577941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5578276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:52:34.5578771Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5579389Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5579907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5580416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5580798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:34.5581129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:52:34.5581467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:34.5581808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:52:34.5582100Z dist init r=0, world=4
2023-01-11T20:52:34.5582282Z dist init r=2, world=4
2023-01-11T20:52:34.5582456Z dist init r=3, world=4
2023-01-11T20:52:34.5582617Z dist init r=1, world=4
2023-01-11T20:52:34.5582819Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T20:52:34.5583273Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10643
2023-01-11T20:52:34.5583754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10644
2023-01-11T20:52:34.5584069Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10645
2023-01-11T20:52:34.5584388Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10646
2023-01-11T20:52:34.5584837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5585155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5585573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5585913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5586244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:34.5586696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5587017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5587437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5587775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5588097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:34.5588561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5588882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5589290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5589622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5589948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:52:34.5590435Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5590912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5591234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5591689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5592025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5592339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:52:34.5592831Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5593346Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5593848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5594229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:52:34.5594611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:34.5595007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:34.5595583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:52:34.5595874Z dist init r=3, world=4
2023-01-11T20:52:34.5596057Z dist init r=1, world=4
2023-01-11T20:52:34.5596222Z dist init r=2, world=4
2023-01-11T20:52:34.5596399Z dist init r=0, world=4
2023-01-11T20:52:34.5596606Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T20:52:34.5597052Z   test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10699
2023-01-11T20:52:34.5597538Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10700
2023-01-11T20:52:34.5597872Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10701
2023-01-11T20:52:34.5598198Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10702
2023-01-11T20:52:34.5598642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5598975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5599406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5599749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5600072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:34.5600539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5600871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5601282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5601621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5601955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:52:34.5602418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5602727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5603147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5603483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5603815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:52:34.5604345Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5604833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5605153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5605565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5605908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5606237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:34.5606726Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5607270Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5607786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5608165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:52:34.5608506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:52:34.5608832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:34.5609163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:34.5609417Z dist init r=3, world=4
2023-01-11T20:52:34.5609582Z dist init r=2, world=4
2023-01-11T20:52:34.5609758Z dist init r=1, world=4
2023-01-11T20:52:34.5609930Z dist init r=0, world=4
2023-01-11T20:52:34.5610119Z skip: Need at least 2 CUDA devices (1.819s)
2023-01-11T20:52:34.5610569Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10755
2023-01-11T20:52:34.5611196Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10756
2023-01-11T20:52:34.5611525Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10757
2023-01-11T20:52:34.5611835Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10758
2023-01-11T20:52:34.5612283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5612609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5613033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5613363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5613699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:34.5614162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5614469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5614887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5615225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5615553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:34.5616032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5616524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5616997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5617421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5617747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5618076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:52:34.5618537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5618845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5619265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5619603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5619975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:52:34.5620454Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5620969Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5621478Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5621857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:52:34.5622186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:34.5622520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:52:34.5622857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:34.5623104Z dist init r=2, world=4
2023-01-11T20:52:34.5623284Z dist init r=1, world=4
2023-01-11T20:52:34.5623464Z dist init r=3, world=4
2023-01-11T20:52:34.5623623Z dist init r=0, world=4
2023-01-11T20:52:34.5623824Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T20:52:34.5624270Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10811
2023-01-11T20:52:34.5624751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10812
2023-01-11T20:52:34.5625060Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10813
2023-01-11T20:52:34.5625373Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10814
2023-01-11T20:52:34.5625817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5626136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5626555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5626899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5627229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:34.5627679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5627998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5628416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5628751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5629070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:52:34.5629565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5629882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5630293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5630626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5630953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:34.5631438Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5631915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5632272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5632696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5633032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5633347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:52:34.5633839Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5634348Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5634846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5635221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:34.5635653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:34.5635993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:52:34.5636320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:52:34.5636573Z dist init r=1, world=4
2023-01-11T20:52:34.5636752Z dist init r=0, world=4
2023-01-11T20:52:34.5636913Z dist init r=3, world=4
2023-01-11T20:52:34.5637087Z dist init r=2, world=4
2023-01-11T20:52:34.5637288Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T20:52:34.5637741Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10867
2023-01-11T20:52:34.5638211Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10868
2023-01-11T20:52:34.5638541Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10869
2023-01-11T20:52:34.5638864Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10870
2023-01-11T20:52:34.5639298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5639621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5640042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5640381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5640700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:34.5641160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5641483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5641955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5642291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5642621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:52:34.5643085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5643392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5643817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5644156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5644486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:34.5644967Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5645293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5645714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5646037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5646364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:52:34.5646851Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5647369Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5647868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5648383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5648767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:34.5649111Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:34.5649442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:52:34.5649779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:52:34.5650035Z dist init r=0, world=4
2023-01-11T20:52:34.5650207Z dist init r=1, world=4
2023-01-11T20:52:34.5650385Z dist init r=3, world=4
2023-01-11T20:52:34.5650563Z dist init r=2, world=4
2023-01-11T20:52:34.5650754Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T20:52:34.5651366Z   test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True (__main__.TestFreezingWeights) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10923
2023-01-11T20:52:34.5651850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10924
2023-01-11T20:52:34.5652180Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10925
2023-01-11T20:52:34.5652490Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10926
2023-01-11T20:52:34.5652945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5653274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5653701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5654032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5654370Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:34.5654901Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5655214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5655638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5655981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5656313Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:52:34.5656827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5657151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5657613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5657960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5658276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:52:34.5658743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:34.5659067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:34.5659475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:34.5659814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:34.5660146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:34.5660636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5661139Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5661653Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5662165Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:52:34.5662545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:34.5662877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:34.5663216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:52:34.5663552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:52:34.5663795Z dist init r=0, world=4
2023-01-11T20:52:34.5663979Z dist init r=1, world=4
2023-01-11T20:52:34.5664154Z dist init r=3, world=4
2023-01-11T20:52:34.5664317Z dist init r=2, world=4
2023-01-11T20:52:34.5664525Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T20:52:34.5664659Z 
2023-01-11T20:52:34.5664858Z ----------------------------------------------------------------------
2023-01-11T20:52:34.5665094Z Ran 8 tests in 14.104s
2023-01-11T20:52:34.5665191Z 
2023-01-11T20:52:34.5665263Z OK (skipped=8)
2023-01-11T20:52:34.5665367Z 
2023-01-11T20:52:34.5665449Z Generating XML reports...
2023-01-11T20:52:34.5665920Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20230111205220.xml
2023-01-11T20:52:34.5666194Z 
2023-01-11T20:52:34.5666474Z ##[endgroup]
2023-01-11T20:52:34.5666955Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_freezing_weights (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_freezing_weights_i3b12so4)
2023-01-11T20:52:34.5667273Z 
2023-01-11T20:52:34.5667472Z Running distributed/algorithms/test_join ... [2023-01-11 20:52:34.553798]
2023-01-11T20:52:34.5667981Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/test_join.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:52:34.554245]
2023-01-11T20:52:50.3658107Z 
2023-01-11T20:52:50.3658829Z Expand the folded group to see the log file of distributed/algorithms/test_join
2023-01-11T20:52:50.3659943Z ##[group]PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-test_join_w07dw81_)
2023-01-11T20:52:50.3660820Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.test_join
2023-01-11T20:52:50.3661124Z 
2023-01-11T20:52:50.3661227Z Running tests...
2023-01-11T20:52:50.3661661Z ----------------------------------------------------------------------
2023-01-11T20:52:50.3662058Z   test_join_kwargs (__main__.TestJoin)
2023-01-11T20:52:50.3662829Z Tests passing keyword arguments to the context manager. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10990
2023-01-11T20:52:50.3663454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10991
2023-01-11T20:52:50.3664212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:50.3664772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:50.3666153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:50.3666680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:50.3667158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:50.3667856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:50.3668404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:50.3669098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:50.3669638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:50.3670136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:50.3670753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:50.3671371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:50.3672157Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:50.3672892Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:50.3673319Z ok (1.868s)
2023-01-11T20:52:50.3673635Z   test_multiple_joinable_disable (__main__.TestJoin)
2023-01-11T20:52:50.3674184Z Tests ``enable=False`` for multiple :class:`Joinable` s. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11018
2023-01-11T20:52:50.3674831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11019
2023-01-11T20:52:50.3675629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:50.3676201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:50.3676962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:50.3677569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:50.3678125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:50.3679084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:50.3679573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:50.3680212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:50.3680709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:50.3681174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:50.3681727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:50.3682261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:50.3683132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:50.3684134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:50.3684641Z ok (1.612s)
2023-01-11T20:52:50.3684977Z   test_multiple_joinables (__main__.TestJoin)
2023-01-11T20:52:50.3685793Z Tests the main hooks and post-hooks of multiple :class:`Joinable` s ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11046
2023-01-11T20:52:50.3686492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11047
2023-01-11T20:52:50.3687258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:50.3687778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:50.3688470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:50.3689040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:50.3689550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:50.3690333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:50.3691060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:50.3691741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:50.3692288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:50.3692742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:50.3693272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:50.3693790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:50.3694546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:50.3695332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:50.3695779Z ok (1.512s)
2023-01-11T20:52:50.3696102Z   test_multiple_joinables_throw (__main__.TestJoin)
2023-01-11T20:52:50.3696589Z Tests ``throw_on_early_termination=True`` for multiple ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11074
2023-01-11T20:52:50.3702838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11075
2023-01-11T20:52:50.3703552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:50.3703906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:50.3704392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:50.3704861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:50.3705250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:50.3705765Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:50.3706278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:50.3707071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:50.3707417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:50.3707790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:50.3708150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:50.3708642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:50.3709192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:50.3709703Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:50.3710058Z ok (1.612s)
2023-01-11T20:52:50.3710366Z   test_single_joinable (__main__.TestJoin)
2023-01-11T20:52:50.3710842Z Tests the main hooks and post-hooks of a single :class:`Joinable` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11102
2023-01-11T20:52:50.3711224Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11103
2023-01-11T20:52:50.3711674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:50.3712004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:50.3712420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:50.3712762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:50.3713234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:50.3714031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:50.3714443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:50.3714871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:50.3715211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:50.3715529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:50.3716050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:50.3716652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:50.3717460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:50.3717968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:50.3718251Z ok (1.612s)
2023-01-11T20:52:50.3718459Z   test_single_joinable_disable (__main__.TestJoin)
2023-01-11T20:52:50.3718808Z Tests ``enable=False`` for a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11130
2023-01-11T20:52:50.3719173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11131
2023-01-11T20:52:50.3719625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:50.3720013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:50.3720426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:50.3720768Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:50.3721085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:50.3721537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:50.3721848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:50.3722270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:50.3722611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:50.3722968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:50.3723316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:50.3723678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:50.3724169Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:50.3724666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:50.3724951Z ok (1.513s)
2023-01-11T20:52:50.3725160Z   test_single_joinable_main_hooks (__main__.TestJoin)
2023-01-11T20:52:50.3725508Z Tests the main hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11158
2023-01-11T20:52:50.3725872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11159
2023-01-11T20:52:50.3726319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:50.3726648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:50.3727057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:50.3727399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:50.3727720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:50.3728178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:50.3728486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:50.3728910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:50.3729254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:50.3729597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:50.3729943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:50.3730307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:50.3730795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:50.3731502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:50.3731773Z ok (1.512s)
2023-01-11T20:52:50.3731981Z   test_single_joinable_post_hooks (__main__.TestJoin)
2023-01-11T20:52:50.3732432Z Tests the post-hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11186
2023-01-11T20:52:50.3732865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11187
2023-01-11T20:52:50.3733308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:50.3733631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:50.3734055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:50.3734383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:50.3734700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:50.3735155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:50.3735464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:50.3735929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:50.3736269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:50.3736585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:50.3737038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:50.3737400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:50.3737893Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:50.3738407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:50.3738679Z ok (1.512s)
2023-01-11T20:52:50.3738884Z   test_single_joinable_throw (__main__.TestJoin)
2023-01-11T20:52:50.3739231Z Tests ``throw_on_early_termination=True`` for a single ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11214
2023-01-11T20:52:50.3739592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11215
2023-01-11T20:52:50.3740036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:50.3740360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:50.3740781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:50.3741109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:50.3741426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:52:50.3741882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:52:50.3742193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:52:50.3742615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:52:50.3742953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:52:50.3743270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:52:50.3743618Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:52:50.3743982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:52:50.3744469Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:50.3744986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:52:50.3745301Z ok (1.512s)
2023-01-11T20:52:50.3745404Z 
2023-01-11T20:52:50.3745602Z ----------------------------------------------------------------------
2023-01-11T20:52:50.3745843Z Ran 9 tests in 14.268s
2023-01-11T20:52:50.3745958Z 
2023-01-11T20:52:50.3746006Z OK
2023-01-11T20:52:50.3746096Z 
2023-01-11T20:52:50.3746179Z Generating XML reports...
2023-01-11T20:52:50.3746606Z Generated XML report: test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20230111205235.xml
2023-01-11T20:52:50.3746846Z 
2023-01-11T20:52:50.3747160Z ##[endgroup]
2023-01-11T20:52:50.3747607Z FINISHED PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-test_join_w07dw81_)
2023-01-11T20:52:50.3747865Z 
2023-01-11T20:52:50.3748076Z Running distributed/tensor/parallel/test_tp_style ... [2023-01-11 20:52:50.366289]
2023-01-11T20:52:50.3748652Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_style.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:52:50.366714]
2023-01-11T20:53:07.5009283Z 
2023-01-11T20:53:07.5009914Z Expand the folded group to see the log file of distributed/tensor/parallel/test_tp_style
2023-01-11T20:53:07.5011374Z ##[group]PRINTING LOG FILE of distributed/tensor/parallel/test_tp_style (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_tp_style_edj0btma)
2023-01-11T20:53:07.5012465Z Test results will be stored in test-reports/python-unittest/distributed.tensor.parallel.test_tp_style
2023-01-11T20:53:07.5012771Z 
2023-01-11T20:53:07.5012884Z Running tests...
2023-01-11T20:53:07.5013354Z ----------------------------------------------------------------------
2023-01-11T20:53:07.5013969Z   test_colwise_parallel_style (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11253
2023-01-11T20:53:07.5014560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11254
2023-01-11T20:53:07.5015062Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11255
2023-01-11T20:53:07.5033024Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11256
2023-01-11T20:53:07.5033573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5033991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5034703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5035279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5035806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:53:07.5036565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5037366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5038081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5038673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5039266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:53:07.5040111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5040578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5041000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5041330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5041643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:53:07.5042300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5042625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5043040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5043387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5043705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:53:07.5044054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:53:07.5044422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:53:07.5044787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:53:07.5045340Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5045725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:53:07.5046203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5046713Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5047222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5047595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T20:53:07.5047957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T20:53:07.5048320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T20:53:07.5048682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T20:53:07.5049154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5049661Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5050049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T20:53:07.5050394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T20:53:07.5051000Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5051514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5051908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T20:53:07.5052254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T20:53:07.5052733Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5053239Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5053746Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5054238Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5054627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T20:53:07.5055045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T20:53:07.5055404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T20:53:07.5055748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T20:53:07.5056232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5056811Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5057199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T20:53:07.5057541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T20:53:07.5058070Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5058581Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5058954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T20:53:07.5059317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T20:53:07.5059793Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5060300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5060792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5061296Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5061579Z ok (2.167s)
2023-01-11T20:53:07.5061909Z   test_make_input_replicate_1d (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11333
2023-01-11T20:53:07.5062297Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11334
2023-01-11T20:53:07.5062621Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11335
2023-01-11T20:53:07.5062943Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11336
2023-01-11T20:53:07.5063390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5063704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5064129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5064472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5064776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:53:07.5065226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5065548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5065966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5066293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5066617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:53:07.5067072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5067421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5067842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5068178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5068493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:53:07.5068936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5069263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5069684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5070021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5070364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:53:07.5070725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:53:07.5071087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:53:07.5071431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:53:07.5071919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5072310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:53:07.5072794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5073288Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5073801Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5074187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T20:53:07.5074543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T20:53:07.5074884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T20:53:07.5075364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5075752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T20:53:07.5076232Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5076730Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5077117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T20:53:07.5077472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T20:53:07.5077940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5078321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T20:53:07.5078671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T20:53:07.5079153Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5079644Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5080065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T20:53:07.5080543Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5081047Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5081420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T20:53:07.5081780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T20:53:07.5082137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T20:53:07.5082622Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5083148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5083536Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T20:53:07.5083893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T20:53:07.5084357Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5084866Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5085249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T20:53:07.5085605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T20:53:07.5086082Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5086587Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5087098Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5087601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5087869Z ok (1.919s)
2023-01-11T20:53:07.5088192Z   test_make_input_shard_1d (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11413
2023-01-11T20:53:07.5088588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11414
2023-01-11T20:53:07.5088910Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11415
2023-01-11T20:53:07.5089219Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11416
2023-01-11T20:53:07.5089671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5090002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5090411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5090748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5091269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:53:07.5091732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5092043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5092466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5092877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5093182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:53:07.5093639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5093958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5094381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5094708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5095024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:53:07.5095476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5095842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5096254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5096590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5096978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:53:07.5097328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:53:07.5097694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:53:07.5098051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:53:07.5098412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:53:07.5098892Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5099411Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5099923Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5100429Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5100800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T20:53:07.5101155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T20:53:07.5101509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T20:53:07.5101865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T20:53:07.5102340Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5102848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5103237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T20:53:07.5103581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T20:53:07.5104060Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5104566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5104954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T20:53:07.5105336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T20:53:07.5105823Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5106326Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5106828Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5107319Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5107704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T20:53:07.5108056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T20:53:07.5108459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T20:53:07.5108800Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T20:53:07.5109276Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5109778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5110165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T20:53:07.5110506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T20:53:07.5110983Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5111487Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5111862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T20:53:07.5112218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T20:53:07.5112693Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5113201Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5113693Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5114188Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5114465Z ok (1.919s)
2023-01-11T20:53:07.5114797Z   test_make_output_replicate_1d (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11493
2023-01-11T20:53:07.5115182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11494
2023-01-11T20:53:07.5115503Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11495
2023-01-11T20:53:07.5115819Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11496
2023-01-11T20:53:07.5116251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5116576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5116996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5117335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5117641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:53:07.5118164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5118485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5118907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5119230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5119545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:53:07.5120002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5120331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5120772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5121111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5121427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:53:07.5121875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5122186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5122607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5122941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5123244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:53:07.5123602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:53:07.5123970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:53:07.5124329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:53:07.5124805Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5125198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:53:07.5125682Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5126193Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5126690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5126967Z ok (1.819s)
2023-01-11T20:53:07.5127296Z   test_make_output_shard_1d (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11549
2023-01-11T20:53:07.5127676Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11550
2023-01-11T20:53:07.5127999Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11551
2023-01-11T20:53:07.5128313Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11552
2023-01-11T20:53:07.5128760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5129072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5129494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5129836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5130145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:53:07.5130635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5131133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5131561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5131890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5132211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:53:07.5132662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5132987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5133454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5133796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5134114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:53:07.5134555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5134880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5135303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5135645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5135948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:53:07.5136309Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:53:07.5136739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:53:07.5137092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:53:07.5137582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5138095Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5138485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:53:07.5138958Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5139471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5139754Z ok (1.819s)
2023-01-11T20:53:07.5140090Z   test_make_output_tensor (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11605
2023-01-11T20:53:07.5140469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11606
2023-01-11T20:53:07.5140794Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11607
2023-01-11T20:53:07.5141116Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11608
2023-01-11T20:53:07.5141551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5141877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5142297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5142634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5142940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:53:07.5143447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5143770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5144250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5144577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5144894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:53:07.5145348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5145658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5146119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5146463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5146781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:53:07.5147222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5147544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5147966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5148293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5148617Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:53:07.5148977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:53:07.5149348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:53:07.5149697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:53:07.5150061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:53:07.5150552Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5151069Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5151574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5152084Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5152371Z ok (1.819s)
2023-01-11T20:53:07.5152694Z   test_prepare_output_error (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11661
2023-01-11T20:53:07.5153097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11662
2023-01-11T20:53:07.5153424Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11663
2023-01-11T20:53:07.5153748Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11664
2023-01-11T20:53:07.5154184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5154511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5154937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5155281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5155590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:53:07.5156084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5156407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5156818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5157159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5157481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:53:07.5157936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5158242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5158699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5159048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5159351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:53:07.5159799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5160124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5160548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5160877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5161195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:53:07.5161547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:53:07.5161911Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:53:07.5162254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:53:07.5162609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:53:07.5163093Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5163595Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5164107Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5164613Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5165005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T20:53:07.5165355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T20:53:07.5165709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T20:53:07.5166069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T20:53:07.5166554Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5166927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T20:53:07.5167406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5167789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T20:53:07.5168307Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5168802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5169186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T20:53:07.5169543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T20:53:07.5170026Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5170515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5171206Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5171650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T20:53:07.5172127Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5172515Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T20:53:07.5172873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T20:53:07.5173234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T20:53:07.5173704Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5174091Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T20:53:07.5174572Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5174961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T20:53:07.5175427Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5175932Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5176316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T20:53:07.5176831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T20:53:07.5177310Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5177821Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5178332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5178836Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5179205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3
2023-01-11T20:53:07.5179559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1
2023-01-11T20:53:07.5179914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0
2023-01-11T20:53:07.5180258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2
2023-01-11T20:53:07.5180741Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 4 nodes.
2023-01-11T20:53:07.5181307Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 4 nodes.
2023-01-11T20:53:07.5181695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0
2023-01-11T20:53:07.5182041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2
2023-01-11T20:53:07.5182518Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 4 nodes.
2023-01-11T20:53:07.5183029Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 4 nodes.
2023-01-11T20:53:07.5183419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1
2023-01-11T20:53:07.5183763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3
2023-01-11T20:53:07.5184273Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 4 nodes.
2023-01-11T20:53:07.5184665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3
2023-01-11T20:53:07.5185147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 4 nodes.
2023-01-11T20:53:07.5185639Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 4 nodes.
2023-01-11T20:53:07.5186142Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 4 nodes.
2023-01-11T20:53:07.5186524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2
2023-01-11T20:53:07.5186880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0
2023-01-11T20:53:07.5187222Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1
2023-01-11T20:53:07.5187705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 4 nodes.
2023-01-11T20:53:07.5188211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 4 nodes.
2023-01-11T20:53:07.5188585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1
2023-01-11T20:53:07.5188941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0
2023-01-11T20:53:07.5189423Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 4 nodes.
2023-01-11T20:53:07.5189929Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 4 nodes.
2023-01-11T20:53:07.5190304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2
2023-01-11T20:53:07.5190658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3
2023-01-11T20:53:07.5191139Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 4 nodes.
2023-01-11T20:53:07.5191645Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 4 nodes.
2023-01-11T20:53:07.5192137Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 4 nodes.
2023-01-11T20:53:07.5192523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3
2023-01-11T20:53:07.5192999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 4 nodes.
2023-01-11T20:53:07.5193385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1
2023-01-11T20:53:07.5193768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0
2023-01-11T20:53:07.5194133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2
2023-01-11T20:53:07.5194618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 4 nodes.
2023-01-11T20:53:07.5195125Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 4 nodes.
2023-01-11T20:53:07.5195499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0
2023-01-11T20:53:07.5195861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2
2023-01-11T20:53:07.5196343Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 4 nodes.
2023-01-11T20:53:07.5196903Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 4 nodes.
2023-01-11T20:53:07.5197282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1
2023-01-11T20:53:07.5197642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3
2023-01-11T20:53:07.5198196Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 4 nodes.
2023-01-11T20:53:07.5198704Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 4 nodes.
2023-01-11T20:53:07.5199090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3
2023-01-11T20:53:07.5199561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 4 nodes.
2023-01-11T20:53:07.5200064Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 4 nodes.
2023-01-11T20:53:07.5200451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2
2023-01-11T20:53:07.5200812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0
2023-01-11T20:53:07.5201161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1
2023-01-11T20:53:07.5201639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 4 nodes.
2023-01-11T20:53:07.5202144Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 4 nodes.
2023-01-11T20:53:07.5202532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0
2023-01-11T20:53:07.5202884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1
2023-01-11T20:53:07.5203365Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 4 nodes.
2023-01-11T20:53:07.5203867Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 4 nodes.
2023-01-11T20:53:07.5204253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2
2023-01-11T20:53:07.5204599Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3
2023-01-11T20:53:07.5205075Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 4 nodes.
2023-01-11T20:53:07.5205577Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 4 nodes.
2023-01-11T20:53:07.5206082Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 4 nodes.
2023-01-11T20:53:07.5206604Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 4 nodes.
2023-01-11T20:53:07.5206888Z ok (2.120s)
2023-01-11T20:53:07.5207219Z   test_rowwise_parallel_style (__main__.TensorParallelStyleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11789
2023-01-11T20:53:07.5207605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11790
2023-01-11T20:53:07.5207927Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11791
2023-01-11T20:53:07.5208243Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11792
2023-01-11T20:53:07.5208690Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5209008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5209472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5209816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5210137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:53:07.5210578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5211073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5211502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5211831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5212152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:53:07.5212610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5212937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5213350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5213692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5214013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:53:07.5214453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:07.5214778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:07.5215200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:07.5215542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:07.5215848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:53:07.5216213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:53:07.5216576Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:53:07.5217002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:53:07.5217348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:53:07.5217833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5218351Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5218853Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5219304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T20:53:07.5219663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T20:53:07.5220145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:07.5220520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T20:53:07.5220875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T20:53:07.5221356Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5221865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5222282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T20:53:07.5222641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T20:53:07.5223120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5223630Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:07.5224004Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T20:53:07.5224362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T20:53:07.5224842Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5225350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5225722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T20:53:07.5226199Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5226695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:07.5227070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T20:53:07.5227427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T20:53:07.5227783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T20:53:07.5228266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5228759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5229144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T20:53:07.5229504Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T20:53:07.5229979Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5230472Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:07.5230857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T20:53:07.5231213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T20:53:07.5231732Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5232223Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5232725Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5233227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:07.5233609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1
2023-01-11T20:53:07.5233959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3
2023-01-11T20:53:07.5234311Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0
2023-01-11T20:53:07.5234698Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2
2023-01-11T20:53:07.5235166Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 4 nodes.
2023-01-11T20:53:07.5235672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 4 nodes.
2023-01-11T20:53:07.5236054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2
2023-01-11T20:53:07.5236409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0
2023-01-11T20:53:07.5236875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 4 nodes.
2023-01-11T20:53:07.5237376Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 4 nodes.
2023-01-11T20:53:07.5237764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1
2023-01-11T20:53:07.5238119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3
2023-01-11T20:53:07.5238582Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 4 nodes.
2023-01-11T20:53:07.5239090Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 4 nodes.
2023-01-11T20:53:07.5239592Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 4 nodes.
2023-01-11T20:53:07.5239973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3
2023-01-11T20:53:07.5240433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 4 nodes.
2023-01-11T20:53:07.5240812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2
2023-01-11T20:53:07.5241165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0
2023-01-11T20:53:07.5241517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1
2023-01-11T20:53:07.5241984Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 4 nodes.
2023-01-11T20:53:07.5242487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 4 nodes.
2023-01-11T20:53:07.5242989Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 4 nodes.
2023-01-11T20:53:07.5243356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1
2023-01-11T20:53:07.5243709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0
2023-01-11T20:53:07.5244222Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 4 nodes.
2023-01-11T20:53:07.5244609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2
2023-01-11T20:53:07.5244955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3
2023-01-11T20:53:07.5245431Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 4 nodes.
2023-01-11T20:53:07.5245931Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 4 nodes.
2023-01-11T20:53:07.5246434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 4 nodes.
2023-01-11T20:53:07.5246953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 4 nodes.
2023-01-11T20:53:07.5247239Z ok (2.020s)
2023-01-11T20:53:07.5247341Z 
2023-01-11T20:53:07.5247542Z ----------------------------------------------------------------------
2023-01-11T20:53:07.5247780Z Ran 8 tests in 15.603s
2023-01-11T20:53:07.5247882Z 
2023-01-11T20:53:07.5247942Z OK
2023-01-11T20:53:07.5248033Z 
2023-01-11T20:53:07.5248117Z Generating XML reports...
2023-01-11T20:53:07.5248609Z Generated XML report: test-reports/python-unittest/distributed.tensor.parallel.test_tp_style/TEST-TensorParallelStyleTest-20230111205251.xml
2023-01-11T20:53:07.5248899Z 
2023-01-11T20:53:07.5249214Z ##[endgroup]
2023-01-11T20:53:07.5249832Z FINISHED PRINTING LOG FILE of distributed/tensor/parallel/test_tp_style (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_tp_style_edj0btma)
2023-01-11T20:53:07.5250118Z 
2023-01-11T20:53:07.5250326Z Running distributed/_tensor/test_redistribute ... [2023-01-11 20:53:07.502232]
2023-01-11T20:53:07.5250975Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_redistribute.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:53:07.502705]
2023-01-11T20:53:24.2458719Z 
2023-01-11T20:53:24.2459634Z Expand the folded group to see the log file of distributed/_tensor/test_redistribute
2023-01-11T20:53:24.2460811Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_redistribute (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_redistribute_wa_lc_4i)
2023-01-11T20:53:24.2461828Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_redistribute
2023-01-11T20:53:24.2462205Z 
2023-01-11T20:53:24.2462311Z Running tests...
2023-01-11T20:53:24.2462840Z ----------------------------------------------------------------------
2023-01-11T20:53:24.2464399Z   test_multi_dim_mesh (__main__.MultiDimRedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11904
2023-01-11T20:53:24.2465169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11905
2023-01-11T20:53:24.2465780Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11906
2023-01-11T20:53:24.2466348Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11907
2023-01-11T20:53:24.2466918Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 11908
2023-01-11T20:53:24.2467427Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 11909
2023-01-11T20:53:24.2467960Z INFO:torch.testing._internal.common_distributed:Started process 6 with pid 11910
2023-01-11T20:53:24.2468479Z INFO:torch.testing._internal.common_distributed:Started process 7 with pid 11911
2023-01-11T20:53:24.2469374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2469964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2470768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2471448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2471762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:53:24.2472257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2472682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2473111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2473437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2473759Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 6
2023-01-11T20:53:24.2474307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2474640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2475048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2475398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2475719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:53:24.2476166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2476488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2477017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2477359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2477666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:53:24.2478128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2478451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2478863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2479203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2479518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:53:24.2479972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2480280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2480708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2481049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2481361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:53:24.2481801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2482123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2482539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2482865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2483182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 7
2023-01-11T20:53:24.2483635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2484040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2484449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2484790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2485106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:53:24.2485454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:53:24.2485821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:53:24.2486180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:53:24.2486539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:53:24.2486921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:53:24.2487280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 6
2023-01-11T20:53:24.2487630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:53:24.2487982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 7
2023-01-11T20:53:24.2488463Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:53:24.2488978Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:53:24.2489492Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:53:24.2489875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 7
2023-01-11T20:53:24.2490225Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2023-01-11T20:53:24.2490715Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:53:24.2491609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:53:24.2492117Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:53:24.2492625Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:53:24.2493136Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 8 nodes.
2023-01-11T20:53:24.2493524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T20:53:24.2493873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T20:53:24.2494226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T20:53:24.2494581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T20:53:24.2494930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2023-01-11T20:53:24.2495266Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 6
2023-01-11T20:53:24.2495745Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:53:24.2496258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:53:24.2496826Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:53:24.2497396Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:53:24.2497902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:53:24.2498409Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:53:24.2498916Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:53:24.2499287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 6
2023-01-11T20:53:24.2499648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T20:53:24.2500054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4
2023-01-11T20:53:24.2500524Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 8 nodes.
2023-01-11T20:53:24.2500910Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T20:53:24.2501303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T20:53:24.2501657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T20:53:24.2502006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5
2023-01-11T20:53:24.2502343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 7
2023-01-11T20:53:24.2502822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:53:24.2503339Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:53:24.2503845Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:53:24.2504217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 7
2023-01-11T20:53:24.2504575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5
2023-01-11T20:53:24.2505060Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:53:24.2505436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T20:53:24.2505912Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:53:24.2506419Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:53:24.2506925Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:53:24.2507412Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 8 nodes.
2023-01-11T20:53:24.2507797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 6
2023-01-11T20:53:24.2508157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4
2023-01-11T20:53:24.2508674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T20:53:24.2509135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T20:53:24.2509828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T20:53:24.2510678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:53:24.2511575Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:53:24.2512408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:53:24.2513070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T20:53:24.2513680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5
2023-01-11T20:53:24.2514273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T20:53:24.2515096Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:53:24.2516020Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:53:24.2516976Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:53:24.2517486Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:53:24.2517866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 6
2023-01-11T20:53:24.2518350Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:4 with 8 nodes.
2023-01-11T20:53:24.2518734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4
2023-01-11T20:53:24.2519081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 7
2023-01-11T20:53:24.2519439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T20:53:24.2519795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T20:53:24.2520275Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:53:24.2520766Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:53:24.2521269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:53:24.2521775Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:53:24.2522280Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:53:24.2522842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3
2023-01-11T20:53:24.2523201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0
2023-01-11T20:53:24.2523567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 7
2023-01-11T20:53:24.2523924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2
2023-01-11T20:53:24.2524390Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:53:24.2524897Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:53:24.2525290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 6
2023-01-11T20:53:24.2525778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 8 nodes.
2023-01-11T20:53:24.2526206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1
2023-01-11T20:53:24.2526562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4
2023-01-11T20:53:24.2526920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5
2023-01-11T20:53:24.2527409Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:53:24.2527904Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:53:24.2528297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 4
2023-01-11T20:53:24.2528656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 5
2023-01-11T20:53:24.2529168Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:53:24.2529679Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:53:24.2530187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:53:24.2530575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1
2023-01-11T20:53:24.2531221Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:53:24.2531731Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:53:24.2532123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0
2023-01-11T20:53:24.2532611Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 8 nodes.
2023-01-11T20:53:24.2532990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 3
2023-01-11T20:53:24.2533352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 2
2023-01-11T20:53:24.2533710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 6
2023-01-11T20:53:24.2534067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 7
2023-01-11T20:53:24.2534540Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:53:24.2535051Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:53:24.2535563Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:53:24.2536070Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:53:24.2536560Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:53:24.2537126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:53:24.2537630Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:53:24.2538133Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:7 with 8 nodes.
2023-01-11T20:53:24.2538715Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:53:24.2539222Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:53:24.2539574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 3
2023-01-11T20:53:24.2540138Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:53:24.2540545Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:53:24.2540897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1
2023-01-11T20:53:24.2541463Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:53:24.2541933Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:53:24.2542274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 7
2023-01-11T20:53:24.2542833Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:53:24.2543251Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:53:24.2543587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 5
2023-01-11T20:53:24.2544142Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:53:24.2544559Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:53:24.2544908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0
2023-01-11T20:53:24.2545456Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:53:24.2545872Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:53:24.2546224Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 4
2023-01-11T20:53:24.2546785Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:53:24.2547187Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:53:24.2547532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 2
2023-01-11T20:53:24.2548088Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:53:24.2548507Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:53:24.2548838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 6
2023-01-11T20:53:24.2549318Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:53:24.2549834Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:53:24.2550346Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:53:24.2550838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:53:24.2551350Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:53:24.2551768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 4
2023-01-11T20:53:24.2552123Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 6
2023-01-11T20:53:24.2552471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0
2023-01-11T20:53:24.2552823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 2
2023-01-11T20:53:24.2553299Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:53:24.2553804Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:53:24.2554343Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:8 with 8 nodes.
2023-01-11T20:53:24.2554732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1
2023-01-11T20:53:24.2555089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 3
2023-01-11T20:53:24.2555430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 7
2023-01-11T20:53:24.2555784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 5
2023-01-11T20:53:24.2556267Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:53:24.2556776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:53:24.2557274Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:53:24.2557781Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:53:24.2558180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 7
2023-01-11T20:53:24.2558542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 5
2023-01-11T20:53:24.2558893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 3
2023-01-11T20:53:24.2559374Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:53:24.2559881Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:53:24.2560393Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:53:24.2560885Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:9 with 8 nodes.
2023-01-11T20:53:24.2561272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 4
2023-01-11T20:53:24.2561633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 6
2023-01-11T20:53:24.2561992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 2
2023-01-11T20:53:24.2562339Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0
2023-01-11T20:53:24.2562822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:53:24.2563209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1
2023-01-11T20:53:24.2563684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:53:24.2564107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0
2023-01-11T20:53:24.2564465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1
2023-01-11T20:53:24.2564946Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:53:24.2565434Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:53:24.2565932Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:53:24.2566321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 4
2023-01-11T20:53:24.2566713Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 6
2023-01-11T20:53:24.2567184Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:53:24.2567686Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:53:24.2568188Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:10 with 8 nodes.
2023-01-11T20:53:24.2568577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 7
2023-01-11T20:53:24.2568922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 5
2023-01-11T20:53:24.2569281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 2
2023-01-11T20:53:24.2569765Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:53:24.2570154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 3
2023-01-11T20:53:24.2570621Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:53:24.2571191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 2
2023-01-11T20:53:24.2571551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 3
2023-01-11T20:53:24.2572027Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:53:24.2572533Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:53:24.2573033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:53:24.2573538Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:53:24.2573912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 6
2023-01-11T20:53:24.2574394Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:53:24.2574781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0
2023-01-11T20:53:24.2575266Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:11 with 8 nodes.
2023-01-11T20:53:24.2575643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 7
2023-01-11T20:53:24.2576000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1
2023-01-11T20:53:24.2576362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 4
2023-01-11T20:53:24.2576849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 5
2023-01-11T20:53:24.2577320Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:53:24.2577708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 5
2023-01-11T20:53:24.2578194Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:53:24.2578702Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:53:24.2579073Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0
2023-01-11T20:53:24.2579606Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:53:24.2580116Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:53:24.2580503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 2
2023-01-11T20:53:24.2580969Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:53:24.2581473Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:53:24.2581976Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:12 with 8 nodes.
2023-01-11T20:53:24.2582349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 3
2023-01-11T20:53:24.2582709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1
2023-01-11T20:53:24.2583072Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 4
2023-01-11T20:53:24.2583435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 6
2023-01-11T20:53:24.2583903Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:53:24.2584286Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 7
2023-01-11T20:53:24.2584767Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:53:24.2585272Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:53:24.2585760Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:53:24.2586256Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:53:24.2586757Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:53:24.2587256Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:53:24.2587743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 8 nodes.
2023-01-11T20:53:24.2588125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 7
2023-01-11T20:53:24.2588489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 3
2023-01-11T20:53:24.2588849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 1
2023-01-11T20:53:24.2589231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 5
2023-01-11T20:53:24.2589598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 0
2023-01-11T20:53:24.2589959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 2
2023-01-11T20:53:24.2590304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 6
2023-01-11T20:53:24.2590665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 4
2023-01-11T20:53:24.2591148Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:14 with 8 nodes.
2023-01-11T20:53:24.2591652Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:14 with 8 nodes.
2023-01-11T20:53:24.2592180Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:14 with 8 nodes.
2023-01-11T20:53:24.2592686Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:14 with 8 nodes.
2023-01-11T20:53:24.2593182Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:14 with 8 nodes.
2023-01-11T20:53:24.2593574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 4
2023-01-11T20:53:24.2594041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:14 with 8 nodes.
2023-01-11T20:53:24.2594430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 6
2023-01-11T20:53:24.2594792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 2
2023-01-11T20:53:24.2595156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 0
2023-01-11T20:53:24.2595623Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:14 with 8 nodes.
2023-01-11T20:53:24.2596135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:14 with 8 nodes.
2023-01-11T20:53:24.2596530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 1
2023-01-11T20:53:24.2596896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 3
2023-01-11T20:53:24.2597246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 5
2023-01-11T20:53:24.2597609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 7
2023-01-11T20:53:24.2598093Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:15 with 8 nodes.
2023-01-11T20:53:24.2598589Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:15 with 8 nodes.
2023-01-11T20:53:24.2599095Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:15 with 8 nodes.
2023-01-11T20:53:24.2599599Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:15 with 8 nodes.
2023-01-11T20:53:24.2600096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:15 with 8 nodes.
2023-01-11T20:53:24.2600586Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:15 with 8 nodes.
2023-01-11T20:53:24.2600970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 3
2023-01-11T20:53:24.2601329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 5
2023-01-11T20:53:24.2601725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 6
2023-01-11T20:53:24.2602071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 7
2023-01-11T20:53:24.2602553Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:15 with 8 nodes.
2023-01-11T20:53:24.2603055Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:15 with 8 nodes.
2023-01-11T20:53:24.2603438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 4
2023-01-11T20:53:24.2603784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 2
2023-01-11T20:53:24.2604140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 0
2023-01-11T20:53:24.2604527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 1
2023-01-11T20:53:24.2605008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:16 with 8 nodes.
2023-01-11T20:53:24.2605380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 1
2023-01-11T20:53:24.2605860Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:16 with 8 nodes.
2023-01-11T20:53:24.2606365Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:16 with 8 nodes.
2023-01-11T20:53:24.2606753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 4
2023-01-11T20:53:24.2607215Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:16 with 8 nodes.
2023-01-11T20:53:24.2607719Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:16 with 8 nodes.
2023-01-11T20:53:24.2608222Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:16 with 8 nodes.
2023-01-11T20:53:24.2608594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 6
2023-01-11T20:53:24.2609074Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:16 with 8 nodes.
2023-01-11T20:53:24.2609580Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:16 with 8 nodes.
2023-01-11T20:53:24.2609968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 5
2023-01-11T20:53:24.2610317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 0
2023-01-11T20:53:24.2610676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 7
2023-01-11T20:53:24.2611210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 2
2023-01-11T20:53:24.2611572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 3
2023-01-11T20:53:24.2612049Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:17 with 8 nodes.
2023-01-11T20:53:24.2612552Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:17 with 8 nodes.
2023-01-11T20:53:24.2612941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 3
2023-01-11T20:53:24.2613299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 2
2023-01-11T20:53:24.2613768Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:17 with 8 nodes.
2023-01-11T20:53:24.2614345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:17 with 8 nodes.
2023-01-11T20:53:24.2614736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 6
2023-01-11T20:53:24.2615218Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:17 with 8 nodes.
2023-01-11T20:53:24.2615705Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:17 with 8 nodes.
2023-01-11T20:53:24.2616088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 1
2023-01-11T20:53:24.2616570Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:17 with 8 nodes.
2023-01-11T20:53:24.2617051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 0
2023-01-11T20:53:24.2617446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 7
2023-01-11T20:53:24.2617934Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:17 with 8 nodes.
2023-01-11T20:53:24.2618323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 4
2023-01-11T20:53:24.2618668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 5
2023-01-11T20:53:24.2619149Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:18 with 8 nodes.
2023-01-11T20:53:24.2619651Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:18 with 8 nodes.
2023-01-11T20:53:24.2620152Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:18 with 8 nodes.
2023-01-11T20:53:24.2620531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 4
2023-01-11T20:53:24.2620892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 5
2023-01-11T20:53:24.2621251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 2
2023-01-11T20:53:24.2621729Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:18 with 8 nodes.
2023-01-11T20:53:24.2622222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:18 with 8 nodes.
2023-01-11T20:53:24.2622719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:18 with 8 nodes.
2023-01-11T20:53:24.2623102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 1
2023-01-11T20:53:24.2623580Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:18 with 8 nodes.
2023-01-11T20:53:24.2623954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 0
2023-01-11T20:53:24.2624435Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:18 with 8 nodes.
2023-01-11T20:53:24.2624819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 3
2023-01-11T20:53:24.2625178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 6
2023-01-11T20:53:24.2625524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 7
2023-01-11T20:53:24.2626005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:19 with 8 nodes.
2023-01-11T20:53:24.2626512Z INFO:torch.distributed.distributed_c10d:Rank 6: Completed store-based barrier for key:store_based_barrier_key:19 with 8 nodes.
2023-01-11T20:53:24.2627033Z INFO:torch.distributed.distributed_c10d:Rank 7: Completed store-based barrier for key:store_based_barrier_key:19 with 8 nodes.
2023-01-11T20:53:24.2627532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:19 with 8 nodes.
2023-01-11T20:53:24.2628028Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:19 with 8 nodes.
2023-01-11T20:53:24.2628533Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:19 with 8 nodes.
2023-01-11T20:53:24.2629017Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:19 with 8 nodes.
2023-01-11T20:53:24.2629515Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:19 with 8 nodes.
2023-01-11T20:53:24.2629792Z ok (4.688s)
2023-01-11T20:53:24.2630162Z   test_partial_to_replicate_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12160
2023-01-11T20:53:24.2630555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12161
2023-01-11T20:53:24.2630880Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12162
2023-01-11T20:53:24.2631203Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12163
2023-01-11T20:53:24.2631651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2631968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2632390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2632732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2633050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:53:24.2633510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2633836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2634258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2634587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2634905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:53:24.2635360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2635670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2636093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2636434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2636752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:53:24.2637192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2637511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2637932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2638270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2638576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:53:24.2638933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:53:24.2639332Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:53:24.2639676Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:53:24.2640031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:53:24.2640515Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2641031Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2641524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2642034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2642319Z ok (1.819s)
2023-01-11T20:53:24.2642676Z   test_partial_to_shard (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12216
2023-01-11T20:53:24.2643045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12217
2023-01-11T20:53:24.2643368Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12218
2023-01-11T20:53:24.2643689Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12219
2023-01-11T20:53:24.2644123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2644448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2644872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2645213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2645521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:53:24.2645976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2646297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2646706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2647045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2647359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:53:24.2647810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2648118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2648547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2648890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2649210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:53:24.2649656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2649982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2650409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2650737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2651322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:53:24.2651862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:53:24.2652369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:53:24.2652716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:53:24.2653221Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2653743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2654135Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:53:24.2654606Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2655122Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2655762Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:53:24.2656190Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:53:24.2656793Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:53:24.2657210Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:53:24.2657757Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:53:24.2658179Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:53:24.2658710Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:53:24.2659128Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:53:24.2659370Z ok (1.719s)
2023-01-11T20:53:24.2659676Z   test_replicate_to_partial (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12272
2023-01-11T20:53:24.2660063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12273
2023-01-11T20:53:24.2660393Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12274
2023-01-11T20:53:24.2660721Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12275
2023-01-11T20:53:24.2661155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2661485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2661916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2662306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2662623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:53:24.2663075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2663398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2663808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2664147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2664462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:53:24.2664925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2665277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2665699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2666041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2666346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:53:24.2666795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2667124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2667544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2667872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2668225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:53:24.2668586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:53:24.2668934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:53:24.2669422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2669816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:53:24.2670172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:53:24.2670644Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2671157Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2671667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2672053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T20:53:24.2672397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T20:53:24.2672749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T20:53:24.2673103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T20:53:24.2673578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:24.2674076Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:24.2674469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T20:53:24.2674825Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T20:53:24.2675308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:24.2675799Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T20:53:24.2676188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T20:53:24.2676668Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:24.2677044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T20:53:24.2677522Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:24.2677942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T20:53:24.2678420Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:24.2678916Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T20:53:24.2679300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T20:53:24.2679656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T20:53:24.2680014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T20:53:24.2680511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:24.2681018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:24.2681524Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:24.2681901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T20:53:24.2682241Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T20:53:24.2682720Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T20:53:24.2683110Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T20:53:24.2683466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T20:53:24.2683935Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:24.2684440Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:24.2684940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:24.2685439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T20:53:24.2685704Z ok (1.821s)
2023-01-11T20:53:24.2686039Z   test_replicate_to_replicate_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12352
2023-01-11T20:53:24.2686444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12353
2023-01-11T20:53:24.2686757Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12354
2023-01-11T20:53:24.2687073Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12355
2023-01-11T20:53:24.2687517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2687845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2688252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2688593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2688911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:53:24.2689349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2689673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2690129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2690469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2690775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:53:24.2691412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2691738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2692148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2692488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2692808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:53:24.2693327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2693642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2694063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2694409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2694728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:53:24.2695078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:53:24.2695440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:53:24.2695926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2696305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:53:24.2696661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:53:24.2697222Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2697737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2698237Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2698519Z ok (1.719s)
2023-01-11T20:53:24.2698854Z   test_replicate_to_shard_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12408
2023-01-11T20:53:24.2699255Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12409
2023-01-11T20:53:24.2699571Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12410
2023-01-11T20:53:24.2699892Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12411
2023-01-11T20:53:24.2700337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2700649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2701068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2701410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2701728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:53:24.2702170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2702492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2702970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2703308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2703614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:53:24.2704066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2704388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2704801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2705137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2705454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:53:24.2705938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2706254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2706675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2707011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2707315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:53:24.2707668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:53:24.2708029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:53:24.2708392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:53:24.2708739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:53:24.2709230Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2709744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2710260Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2710765Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2711054Z ok (1.720s)
2023-01-11T20:53:24.2711388Z   test_shard_to_replicate_forward_backward (__main__.RedistributeTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12464
2023-01-11T20:53:24.2711774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12465
2023-01-11T20:53:24.2712104Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12466
2023-01-11T20:53:24.2712424Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12467
2023-01-11T20:53:24.2712875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2713192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2713617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2713959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2714285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:53:24.2714730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2715055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2715521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2715852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2716171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:53:24.2716629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2716953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2717362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2717706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2718027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:53:24.2718498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:53:24.2718826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:53:24.2719247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:53:24.2719585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:53:24.2719887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:53:24.2720244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:53:24.2720605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:53:24.2720963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:53:24.2721312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:53:24.2721799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2722312Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2722807Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2723311Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:53:24.2723599Z ok (1.719s)
2023-01-11T20:53:24.2723702Z 
2023-01-11T20:53:24.2723902Z ----------------------------------------------------------------------
2023-01-11T20:53:24.2724133Z Ran 7 tests in 15.207s
2023-01-11T20:53:24.2724245Z 
2023-01-11T20:53:24.2724305Z OK
2023-01-11T20:53:24.2724396Z 
2023-01-11T20:53:24.2724481Z Generating XML reports...
2023-01-11T20:53:24.2724944Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-MultiDimRedistributeTest-20230111205308.xml
2023-01-11T20:53:24.2725539Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-RedistributeTest-20230111205308.xml
2023-01-11T20:53:24.2725790Z 
2023-01-11T20:53:24.2726133Z ##[endgroup]
2023-01-11T20:53:24.2726591Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_redistribute (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_redistribute_wa_lc_4i)
2023-01-11T20:53:24.2726840Z 
2023-01-11T20:53:24.2727016Z Running distributed/test_store ... [2023-01-11 20:53:24.247108]
2023-01-11T20:53:24.2727528Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_store.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:53:24.247573]
2023-01-11T20:54:27.4277981Z 
2023-01-11T20:54:27.4281511Z Expand the folded group to see the log file of distributed/test_store
2023-01-11T20:54:27.4292544Z ##[group]PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/workspace/test/test-reports/distributed-test_store_zk19vj5v)
2023-01-11T20:54:27.4293600Z <unittest.suite.TestSuite tests=[<__main__.FileStoreTest testMethod=test_compare_set>, <__main__.FileStoreTest testMethod=test_init_pg_and_rpc_with_same_file>, <__main__.FileStoreTest testMethod=test_refcount>, <__main__.FileStoreTest testMethod=test_set_get>]>
2023-01-11T20:54:27.4294243Z test_compare_set (__main__.FileStoreTest)
2023-01-11T20:54:27.4294646Z test_init_pg_and_rpc_with_same_file (__main__.FileStoreTest)
2023-01-11T20:54:27.4294888Z test_refcount (__main__.FileStoreTest)
2023-01-11T20:54:27.4295092Z test_set_get (__main__.FileStoreTest)
2023-01-11T20:54:27.4295428Z <unittest.suite.TestSuite tests=[<__main__.HashStoreTest testMethod=test_compare_set>, <__main__.HashStoreTest testMethod=test_set_get>]>
2023-01-11T20:54:27.4295749Z test_compare_set (__main__.HashStoreTest)
2023-01-11T20:54:27.4296123Z test_set_get (__main__.HashStoreTest)
2023-01-11T20:54:27.4296490Z <unittest.suite.TestSuite tests=[<__main__.PrefixFileStoreTest testMethod=test_compare_set>, <__main__.PrefixFileStoreTest testMethod=test_set_get>]>
2023-01-11T20:54:27.4312090Z test_compare_set (__main__.PrefixFileStoreTest)
2023-01-11T20:54:27.4312367Z test_set_get (__main__.PrefixFileStoreTest)
2023-01-11T20:54:27.4312694Z <unittest.suite.TestSuite tests=[<__main__.PrefixStoreTest testMethod=test_get_underlying_store>]>
2023-01-11T20:54:27.4313002Z test_get_underlying_store (__main__.PrefixStoreTest)
2023-01-11T20:54:27.4313390Z <unittest.suite.TestSuite tests=[<__main__.PrefixTCPStoreTest testMethod=test_compare_set>, <__main__.PrefixTCPStoreTest testMethod=test_set_get>]>
2023-01-11T20:54:27.4313753Z test_compare_set (__main__.PrefixTCPStoreTest)
2023-01-11T20:54:27.4314000Z test_set_get (__main__.PrefixTCPStoreTest)
2023-01-11T20:54:27.4314281Z <unittest.suite.TestSuite tests=[<__main__.PythonStoreTest testMethod=test_set_get>]>
2023-01-11T20:54:27.4314575Z test_set_get (__main__.PythonStoreTest)
2023-01-11T20:54:27.4314868Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_nominal>]>
2023-01-11T20:54:27.4315144Z test_nominal (__main__.RendezvousEnvTest)
2023-01-11T20:54:27.4315512Z <unittest.suite.TestSuite tests=[<__main__.RendezvousFileTest testMethod=test_common_errors>, <__main__.RendezvousFileTest testMethod=test_nominal>]>
2023-01-11T20:54:27.4315870Z test_common_errors (__main__.RendezvousFileTest)
2023-01-11T20:54:27.4316114Z test_nominal (__main__.RendezvousFileTest)
2023-01-11T20:54:27.4316594Z <unittest.suite.TestSuite tests=[<__main__.RendezvousTCPTest testMethod=test_common_errors>, <__main__.RendezvousTCPTest testMethod=test_dns_timeout>, <__main__.RendezvousTCPTest testMethod=test_nominal>, <__main__.RendezvousTCPTest testMethod=test_tcp_store_timeout_set>]>
2023-01-11T20:54:27.4317074Z test_common_errors (__main__.RendezvousTCPTest)
2023-01-11T20:54:27.4317323Z test_dns_timeout (__main__.RendezvousTCPTest)
2023-01-11T20:54:27.4317548Z test_nominal (__main__.RendezvousTCPTest)
2023-01-11T20:54:27.4317796Z test_tcp_store_timeout_set (__main__.RendezvousTCPTest)
2023-01-11T20:54:27.4318178Z <unittest.suite.TestSuite tests=[<__main__.RendezvousTest testMethod=test_unknown_handler>, <__main__.RendezvousTest testMethod=test_url_with_node_params>]>
2023-01-11T20:54:27.4318533Z test_unknown_handler (__main__.RendezvousTest)
2023-01-11T20:54:27.4318765Z test_url_with_node_params (__main__.RendezvousTest)
2023-01-11T20:54:27.4319502Z <unittest.suite.TestSuite tests=[<__main__.TCPStoreTest testMethod=test_address_already_in_use>, <__main__.TCPStoreTest testMethod=test_compare_set>, <__main__.TCPStoreTest testMethod=test_init_pg_and_rpc_with_same_socket>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_fixed_world_size>, <__main__.TCPStoreTest testMethod=test_multi_worker_with_nonfixed_world_size>, <__main__.TCPStoreTest testMethod=test_multitenancy>, <__main__.TCPStoreTest testMethod=test_numkeys_delkeys>, <__main__.TCPStoreTest testMethod=test_set_get>]>
2023-01-11T20:54:27.4320349Z test_address_already_in_use (__main__.TCPStoreTest)
2023-01-11T20:54:27.4320584Z test_compare_set (__main__.TCPStoreTest)
2023-01-11T20:54:27.4320823Z test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest)
2023-01-11T20:54:27.4321095Z test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest)
2023-01-11T20:54:27.4321373Z test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest)
2023-01-11T20:54:27.4321627Z test_multitenancy (__main__.TCPStoreTest)
2023-01-11T20:54:27.4321848Z test_numkeys_delkeys (__main__.TCPStoreTest)
2023-01-11T20:54:27.4322070Z test_set_get (__main__.TCPStoreTest)
2023-01-11T20:54:27.4322519Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4322984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4323364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4323797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4324139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4324289Z 
2023-01-11T20:54:27.4324362Z Running tests...
2023-01-11T20:54:27.4324664Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4324950Z   test_compare_set (__main__.FileStoreTest) ... ok (0.247s)
2023-01-11T20:54:27.4325100Z 
2023-01-11T20:54:27.4325281Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4325516Z Ran 1 test in 0.247s
2023-01-11T20:54:27.4325627Z 
2023-01-11T20:54:27.4325688Z OK
2023-01-11T20:54:27.4325778Z 
2023-01-11T20:54:27.4325849Z Generating XML reports...
2023-01-11T20:54:27.4326261Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20230111205326.xml
2023-01-11T20:54:27.4326719Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4327172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4327487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4327909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4328248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4328409Z 
2023-01-11T20:54:27.4328479Z Running tests...
2023-01-11T20:54:27.4328764Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4329169Z   test_init_pg_and_rpc_with_same_file (__main__.FileStoreTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:54:27.4329718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T20:54:27.4329995Z ok (0.291s)
2023-01-11T20:54:27.4330094Z 
2023-01-11T20:54:27.4330287Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4330523Z Ran 1 test in 0.291s
2023-01-11T20:54:27.4330633Z 
2023-01-11T20:54:27.4330692Z OK
2023-01-11T20:54:27.4330768Z 
2023-01-11T20:54:27.4331035Z Generating XML reports...
2023-01-11T20:54:27.4331452Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20230111205328.xml
2023-01-11T20:54:27.4331904Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4332342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4332670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4333095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4333510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4333657Z 
2023-01-11T20:54:27.4333728Z Running tests...
2023-01-11T20:54:27.4334026Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4334307Z   test_refcount (__main__.FileStoreTest) ... ok (0.248s)
2023-01-11T20:54:27.4334449Z 
2023-01-11T20:54:27.4334625Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4334855Z Ran 1 test in 0.248s
2023-01-11T20:54:27.4334963Z 
2023-01-11T20:54:27.4335023Z OK
2023-01-11T20:54:27.4335112Z 
2023-01-11T20:54:27.4335194Z Generating XML reports...
2023-01-11T20:54:27.4335589Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20230111205330.xml
2023-01-11T20:54:27.4336036Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4336539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4336855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4337347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4337688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4337850Z 
2023-01-11T20:54:27.4337921Z Running tests...
2023-01-11T20:54:27.4338206Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4338480Z   test_set_get (__main__.FileStoreTest) ... ok (0.251s)
2023-01-11T20:54:27.4338622Z 
2023-01-11T20:54:27.4338812Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4339033Z Ran 1 test in 0.251s
2023-01-11T20:54:27.4339145Z 
2023-01-11T20:54:27.4339204Z OK
2023-01-11T20:54:27.4339296Z 
2023-01-11T20:54:27.4339382Z Generating XML reports...
2023-01-11T20:54:27.4339792Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20230111205331.xml
2023-01-11T20:54:27.4340229Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4340681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4341005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4341413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4341752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4341915Z 
2023-01-11T20:54:27.4341986Z Running tests...
2023-01-11T20:54:27.4342280Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4342551Z   test_compare_set (__main__.HashStoreTest) ... ok (0.248s)
2023-01-11T20:54:27.4342698Z 
2023-01-11T20:54:27.4342890Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4343123Z Ran 1 test in 0.249s
2023-01-11T20:54:27.4343234Z 
2023-01-11T20:54:27.4343281Z OK
2023-01-11T20:54:27.4343369Z 
2023-01-11T20:54:27.4343450Z Generating XML reports...
2023-01-11T20:54:27.4343856Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20230111205333.xml
2023-01-11T20:54:27.4344303Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4344745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4345068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4345488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4345868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4346029Z 
2023-01-11T20:54:27.4346099Z Running tests...
2023-01-11T20:54:27.4346396Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4346668Z   test_set_get (__main__.HashStoreTest) ... ok (0.247s)
2023-01-11T20:54:27.4346798Z 
2023-01-11T20:54:27.4346992Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4347223Z Ran 1 test in 0.247s
2023-01-11T20:54:27.4347333Z 
2023-01-11T20:54:27.4347393Z OK
2023-01-11T20:54:27.4347483Z 
2023-01-11T20:54:27.4347554Z Generating XML reports...
2023-01-11T20:54:27.4348062Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20230111205335.xml
2023-01-11T20:54:27.4348517Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4349010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4349328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4349752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4350093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4350252Z 
2023-01-11T20:54:27.4350310Z Running tests...
2023-01-11T20:54:27.4350605Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4350898Z   test_compare_set (__main__.PrefixFileStoreTest) ... ok (0.250s)
2023-01-11T20:54:27.4351055Z 
2023-01-11T20:54:27.4351248Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4351469Z Ran 1 test in 0.250s
2023-01-11T20:54:27.4351580Z 
2023-01-11T20:54:27.4351641Z OK
2023-01-11T20:54:27.4351731Z 
2023-01-11T20:54:27.4351814Z Generating XML reports...
2023-01-11T20:54:27.4352233Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20230111205337.xml
2023-01-11T20:54:27.4352700Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4353152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4353475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4353882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4354219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4354379Z 
2023-01-11T20:54:27.4354449Z Running tests...
2023-01-11T20:54:27.4354733Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4355019Z   test_set_get (__main__.PrefixFileStoreTest) ... ok (0.250s)
2023-01-11T20:54:27.4355174Z 
2023-01-11T20:54:27.4355368Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4355601Z Ran 1 test in 0.250s
2023-01-11T20:54:27.4355699Z 
2023-01-11T20:54:27.4355758Z OK
2023-01-11T20:54:27.4355846Z 
2023-01-11T20:54:27.4355928Z Generating XML reports...
2023-01-11T20:54:27.4356356Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20230111205338.xml
2023-01-11T20:54:27.4356810Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4357259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4357585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4358007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4358333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4358538Z 
2023-01-11T20:54:27.4358610Z Running tests...
2023-01-11T20:54:27.4358907Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4359188Z   test_get_underlying_store (__main__.PrefixStoreTest) ... ok (0.002s)
2023-01-11T20:54:27.4359346Z 
2023-01-11T20:54:27.4359538Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4359768Z Ran 1 test in 0.002s
2023-01-11T20:54:27.4359875Z 
2023-01-11T20:54:27.4359934Z OK
2023-01-11T20:54:27.4360011Z 
2023-01-11T20:54:27.4360093Z Generating XML reports...
2023-01-11T20:54:27.4360509Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20230111205340.xml
2023-01-11T20:54:27.4360961Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4361400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4361763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4362187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4362527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4362686Z 
2023-01-11T20:54:27.4362745Z Running tests...
2023-01-11T20:54:27.4363037Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4363327Z   test_compare_set (__main__.PrefixTCPStoreTest) ... ok (0.253s)
2023-01-11T20:54:27.4363483Z 
2023-01-11T20:54:27.4363662Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4363893Z Ran 1 test in 0.253s
2023-01-11T20:54:27.4364000Z 
2023-01-11T20:54:27.4364059Z OK
2023-01-11T20:54:27.4364148Z 
2023-01-11T20:54:27.4364231Z Generating XML reports...
2023-01-11T20:54:27.4364648Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20230111205341.xml
2023-01-11T20:54:27.4365109Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4365560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4365869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4366293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4366630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4366792Z 
2023-01-11T20:54:27.4366863Z Running tests...
2023-01-11T20:54:27.4367145Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4367430Z   test_set_get (__main__.PrefixTCPStoreTest) ... ok (0.249s)
2023-01-11T20:54:27.4367580Z 
2023-01-11T20:54:27.4367767Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4367991Z Ran 1 test in 0.249s
2023-01-11T20:54:27.4368100Z 
2023-01-11T20:54:27.4368159Z OK
2023-01-11T20:54:27.4368247Z 
2023-01-11T20:54:27.4368329Z Generating XML reports...
2023-01-11T20:54:27.4368750Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20230111205343.xml
2023-01-11T20:54:27.4369200Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4369648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4369969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4370375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4370710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4371029Z 
2023-01-11T20:54:27.4371101Z Running tests...
2023-01-11T20:54:27.4371476Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4371745Z   test_set_get (__main__.PythonStoreTest) ... ok (0.246s)
2023-01-11T20:54:27.4371890Z 
2023-01-11T20:54:27.4372081Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4372312Z Ran 1 test in 0.246s
2023-01-11T20:54:27.4372422Z 
2023-01-11T20:54:27.4372472Z OK
2023-01-11T20:54:27.4372563Z 
2023-01-11T20:54:27.4372646Z Generating XML reports...
2023-01-11T20:54:27.4373063Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20230111205345.xml
2023-01-11T20:54:27.4373517Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4373954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4374279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4374744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4375083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4375231Z 
2023-01-11T20:54:27.4375301Z Running tests...
2023-01-11T20:54:27.4375597Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4375876Z   test_nominal (__main__.RendezvousEnvTest) ... ok (0.250s)
2023-01-11T20:54:27.4376023Z 
2023-01-11T20:54:27.4376202Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4376432Z Ran 1 test in 0.250s
2023-01-11T20:54:27.4376542Z 
2023-01-11T20:54:27.4376603Z OK
2023-01-11T20:54:27.4376691Z 
2023-01-11T20:54:27.4376761Z Generating XML reports...
2023-01-11T20:54:27.4377266Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20230111205347.xml
2023-01-11T20:54:27.4377730Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4378182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4378495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4378921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4379261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4379421Z 
2023-01-11T20:54:27.4379492Z Running tests...
2023-01-11T20:54:27.4379776Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4380066Z   test_common_errors (__main__.RendezvousFileTest) ... ok (0.246s)
2023-01-11T20:54:27.4380222Z 
2023-01-11T20:54:27.4380414Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4380636Z Ran 1 test in 0.246s
2023-01-11T20:54:27.4380747Z 
2023-01-11T20:54:27.4380811Z OK
2023-01-11T20:54:27.4380900Z 
2023-01-11T20:54:27.4380982Z Generating XML reports...
2023-01-11T20:54:27.4381391Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20230111205348.xml
2023-01-11T20:54:27.4381848Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4382296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4382610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4383035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4383376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4383538Z 
2023-01-11T20:54:27.4383609Z Running tests...
2023-01-11T20:54:27.4383895Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4384216Z   test_nominal (__main__.RendezvousFileTest) ... ok (0.244s)
2023-01-11T20:54:27.4384363Z 
2023-01-11T20:54:27.4384554Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4384769Z Ran 1 test in 0.244s
2023-01-11T20:54:27.4384877Z 
2023-01-11T20:54:27.4384936Z OK
2023-01-11T20:54:27.4385025Z 
2023-01-11T20:54:27.4385107Z Generating XML reports...
2023-01-11T20:54:27.4385517Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20230111205350.xml
2023-01-11T20:54:27.4385981Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4386433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4386756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4387199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4387540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4387701Z 
2023-01-11T20:54:27.4387772Z Running tests...
2023-01-11T20:54:27.4388068Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4388344Z   test_common_errors (__main__.RendezvousTCPTest) ... ok (0.244s)
2023-01-11T20:54:27.4388499Z 
2023-01-11T20:54:27.4388691Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4388923Z Ran 1 test in 0.244s
2023-01-11T20:54:27.4389032Z 
2023-01-11T20:54:27.4389080Z OK
2023-01-11T20:54:27.4389169Z 
2023-01-11T20:54:27.4389250Z Generating XML reports...
2023-01-11T20:54:27.4389669Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20230111205352.xml
2023-01-11T20:54:27.4390110Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4390562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4390888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4391311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4391638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4391796Z 
2023-01-11T20:54:27.4391867Z Running tests...
2023-01-11T20:54:27.4392164Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4392713Z   test_dns_timeout (__main__.RendezvousTCPTest) ... [W socket.cpp:601] [c10d] The IPv6 network addresses of (dnsnotexist, 23456) cannot be retrieved (gai error: -2 - Name or service not known).
2023-01-11T20:54:27.4393137Z [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (dnsnotexist, 23456).
2023-01-11T20:54:27.4393397Z ok (0.250s)
2023-01-11T20:54:27.4393497Z 
2023-01-11T20:54:27.4393690Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4393910Z Ran 1 test in 0.250s
2023-01-11T20:54:27.4394018Z 
2023-01-11T20:54:27.4394076Z OK
2023-01-11T20:54:27.4394165Z 
2023-01-11T20:54:27.4394247Z Generating XML reports...
2023-01-11T20:54:27.4394666Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20230111205353.xml
2023-01-11T20:54:27.4395109Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4395558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4395882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4396291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4396684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4396844Z 
2023-01-11T20:54:27.4396915Z Running tests...
2023-01-11T20:54:27.4397213Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4397486Z   test_nominal (__main__.RendezvousTCPTest) ... ok (0.246s)
2023-01-11T20:54:27.4397635Z 
2023-01-11T20:54:27.4397830Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4398067Z Ran 1 test in 0.246s
2023-01-11T20:54:27.4398178Z 
2023-01-11T20:54:27.4398225Z OK
2023-01-11T20:54:27.4398315Z 
2023-01-11T20:54:27.4398399Z Generating XML reports...
2023-01-11T20:54:27.4398822Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20230111205355.xml
2023-01-11T20:54:27.4399281Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4399758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4400085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4400506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4400831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4400989Z 
2023-01-11T20:54:27.4401059Z Running tests...
2023-01-11T20:54:27.4401355Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4401654Z   test_tcp_store_timeout_set (__main__.RendezvousTCPTest) ... ok (10.317s)
2023-01-11T20:54:27.4401805Z 
2023-01-11T20:54:27.4401996Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4402232Z Ran 1 test in 10.317s
2023-01-11T20:54:27.4402344Z 
2023-01-11T20:54:27.4402406Z OK
2023-01-11T20:54:27.4402497Z 
2023-01-11T20:54:27.4402567Z Generating XML reports...
2023-01-11T20:54:27.4402987Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20230111205357.xml
2023-01-11T20:54:27.4403442Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4403892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4404204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4404624Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4404960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4405118Z 
2023-01-11T20:54:27.4405187Z Running tests...
2023-01-11T20:54:27.4405470Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4405750Z   test_unknown_handler (__main__.RendezvousTest) ... ok (0.246s)
2023-01-11T20:54:27.4405907Z 
2023-01-11T20:54:27.4406105Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4406330Z Ran 1 test in 0.246s
2023-01-11T20:54:27.4406440Z 
2023-01-11T20:54:27.4406502Z OK
2023-01-11T20:54:27.4406595Z 
2023-01-11T20:54:27.4406679Z Generating XML reports...
2023-01-11T20:54:27.4407077Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20230111205409.xml
2023-01-11T20:54:27.4407532Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4407986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4408314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4408727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4409071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4409275Z 
2023-01-11T20:54:27.4409346Z Running tests...
2023-01-11T20:54:27.4409630Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4409925Z   test_url_with_node_params (__main__.RendezvousTest) ... ok (0.247s)
2023-01-11T20:54:27.4410082Z 
2023-01-11T20:54:27.4410278Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4410515Z Ran 1 test in 0.247s
2023-01-11T20:54:27.4410626Z 
2023-01-11T20:54:27.4410674Z OK
2023-01-11T20:54:27.4410765Z 
2023-01-11T20:54:27.4410980Z Generating XML reports...
2023-01-11T20:54:27.4411398Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20230111205410.xml
2023-01-11T20:54:27.4411838Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4412294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4412683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4413109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4413440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4413603Z 
2023-01-11T20:54:27.4413677Z Running tests...
2023-01-11T20:54:27.4413978Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4414510Z   test_address_already_in_use (__main__.TCPStoreTest) ... [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:42495 (errno: 98 - Address already in use).
2023-01-11T20:54:27.4414995Z [W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:42495 (errno: 98 - Address already in use).
2023-01-11T20:54:27.4415336Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address.
2023-01-11T20:54:27.4415581Z ok (0.247s)
2023-01-11T20:54:27.4415687Z 
2023-01-11T20:54:27.4415870Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4416109Z Ran 1 test in 0.247s
2023-01-11T20:54:27.4416218Z 
2023-01-11T20:54:27.4416278Z OK
2023-01-11T20:54:27.4416366Z 
2023-01-11T20:54:27.4416450Z Generating XML reports...
2023-01-11T20:54:27.4416847Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111205412.xml
2023-01-11T20:54:27.4417361Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4417814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4418129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4418558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4418902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4419066Z 
2023-01-11T20:54:27.4419138Z Running tests...
2023-01-11T20:54:27.4419422Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4419701Z   test_compare_set (__main__.TCPStoreTest) ... ok (0.247s)
2023-01-11T20:54:27.4419846Z 
2023-01-11T20:54:27.4420039Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4420257Z Ran 1 test in 0.247s
2023-01-11T20:54:27.4420367Z 
2023-01-11T20:54:27.4420426Z OK
2023-01-11T20:54:27.4420517Z 
2023-01-11T20:54:27.4420600Z Generating XML reports...
2023-01-11T20:54:27.4421005Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111205414.xml
2023-01-11T20:54:27.4421436Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4421889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4422270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4422681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4423022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4423180Z 
2023-01-11T20:54:27.4423252Z Running tests...
2023-01-11T20:54:27.4423550Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4423942Z   test_init_pg_and_rpc_with_same_socket (__main__.TCPStoreTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:54:27.4424494Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T20:54:27.4424778Z ok (0.298s)
2023-01-11T20:54:27.4424877Z 
2023-01-11T20:54:27.4425069Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4425324Z Ran 1 test in 0.298s
2023-01-11T20:54:27.4425434Z 
2023-01-11T20:54:27.4425495Z OK
2023-01-11T20:54:27.4425586Z 
2023-01-11T20:54:27.4425669Z Generating XML reports...
2023-01-11T20:54:27.4426058Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111205415.xml
2023-01-11T20:54:27.4426501Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4426949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4427274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4427682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4428021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4428181Z 
2023-01-11T20:54:27.4428250Z Running tests...
2023-01-11T20:54:27.4428540Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4428841Z   test_multi_worker_with_fixed_world_size (__main__.TCPStoreTest) ... ok (0.250s)
2023-01-11T20:54:27.4429008Z 
2023-01-11T20:54:27.4429197Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4429430Z Ran 1 test in 0.250s
2023-01-11T20:54:27.4429525Z 
2023-01-11T20:54:27.4429588Z OK
2023-01-11T20:54:27.4429676Z 
2023-01-11T20:54:27.4429757Z Generating XML reports...
2023-01-11T20:54:27.4430162Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111205417.xml
2023-01-11T20:54:27.4430595Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4431042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4431363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4431791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4432120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4432281Z 
2023-01-11T20:54:27.4432351Z Running tests...
2023-01-11T20:54:27.4432645Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4432939Z   test_multi_worker_with_nonfixed_world_size (__main__.TCPStoreTest) ... ok (0.248s)
2023-01-11T20:54:27.4433109Z 
2023-01-11T20:54:27.4433297Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4433530Z Ran 1 test in 0.249s
2023-01-11T20:54:27.4433639Z 
2023-01-11T20:54:27.4433698Z OK
2023-01-11T20:54:27.4433775Z 
2023-01-11T20:54:27.4433857Z Generating XML reports...
2023-01-11T20:54:27.4434262Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111205419.xml
2023-01-11T20:54:27.4434744Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4435179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4435501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4435922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4436263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4436411Z 
2023-01-11T20:54:27.4436482Z Running tests...
2023-01-11T20:54:27.4436778Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4437061Z   test_multitenancy (__main__.TCPStoreTest) ... ok (0.250s)
2023-01-11T20:54:27.4437211Z 
2023-01-11T20:54:27.4437389Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4437623Z Ran 1 test in 0.251s
2023-01-11T20:54:27.4437768Z 
2023-01-11T20:54:27.4437828Z OK
2023-01-11T20:54:27.4437917Z 
2023-01-11T20:54:27.4438001Z Generating XML reports...
2023-01-11T20:54:27.4438389Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111205421.xml
2023-01-11T20:54:27.4438834Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4439284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4439595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4440014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4440353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4440512Z 
2023-01-11T20:54:27.4440583Z Running tests...
2023-01-11T20:54:27.4440869Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4441153Z   test_numkeys_delkeys (__main__.TCPStoreTest) ... ok (2.258s)
2023-01-11T20:54:27.4441303Z 
2023-01-11T20:54:27.4441496Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4441715Z Ran 1 test in 2.258s
2023-01-11T20:54:27.4441823Z 
2023-01-11T20:54:27.4441884Z OK
2023-01-11T20:54:27.4441973Z 
2023-01-11T20:54:27.4442055Z Generating XML reports...
2023-01-11T20:54:27.4442458Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111205422.xml
2023-01-11T20:54:27.4442889Z Test results will be stored in test-reports/python-unittest/distributed.test_store
2023-01-11T20:54:27.4443335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:27.4443659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:27.4444071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:27.4444416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:27.4444575Z 
2023-01-11T20:54:27.4444645Z Running tests...
2023-01-11T20:54:27.4444940Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4445197Z   test_set_get (__main__.TCPStoreTest) ... ok (0.250s)
2023-01-11T20:54:27.4445334Z 
2023-01-11T20:54:27.4445523Z ----------------------------------------------------------------------
2023-01-11T20:54:27.4445754Z Ran 1 test in 0.251s
2023-01-11T20:54:27.4445862Z 
2023-01-11T20:54:27.4445909Z OK
2023-01-11T20:54:27.4445998Z 
2023-01-11T20:54:27.4446080Z Generating XML reports...
2023-01-11T20:54:27.4446480Z Generated XML report: test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111205426.xml
2023-01-11T20:54:27.4446706Z 
2023-01-11T20:54:27.4446991Z ##[endgroup]
2023-01-11T20:54:27.4447399Z FINISHED PRINTING LOG FILE of distributed/test_store (/var/lib/jenkins/workspace/test/test-reports/distributed-test_store_zk19vj5v)
2023-01-11T20:54:27.4447667Z 
2023-01-11T20:54:27.4447873Z Running distributed/_composable/test_fully_shard ... [2023-01-11 20:54:27.428468]
2023-01-11T20:54:27.4448403Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_fully_shard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:54:27.428814]
2023-01-11T20:54:48.6338418Z 
2023-01-11T20:54:48.6338972Z Expand the folded group to see the log file of distributed/_composable/test_fully_shard
2023-01-11T20:54:48.6340573Z ##[group]PRINTING LOG FILE of distributed/_composable/test_fully_shard (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_fully_shard_njr6pcid)
2023-01-11T20:54:48.6341627Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_fully_shard
2023-01-11T20:54:48.6341968Z 
2023-01-11T20:54:48.6342176Z Running tests...
2023-01-11T20:54:48.6342963Z ----------------------------------------------------------------------
2023-01-11T20:54:48.6343471Z   test_device_id (__main__.TestFSDPInitialization)
2023-01-11T20:54:48.6344013Z Tests passing a ``device_id``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12956
2023-01-11T20:54:48.6344411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12957
2023-01-11T20:54:48.6344985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6345472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6345935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6346283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6346615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:54:48.6347118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6347600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6347955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6348422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6348823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6349197Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:54:48.6349691Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6350079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:54:48.6350412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:54:48.6350665Z dist init r=0, world=2
2023-01-11T20:54:48.6350842Z dist init r=1, world=2
2023-01-11T20:54:48.6351036Z skip: Need at least 2 CUDA devices (1.862s)
2023-01-11T20:54:48.6351296Z   test_manual_fully_shard (__main__.TestFSDPInitialization)
2023-01-11T20:54:48.6351645Z Tests manually applying ``fully_shard``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12984
2023-01-11T20:54:48.6352009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12985
2023-01-11T20:54:48.6352445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6352774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6353199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6353652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6353988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:54:48.6354456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6354783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6355192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6355540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6355870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:54:48.6356425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6356939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6357323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:54:48.6357671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:54:48.6357914Z dist init r=1, world=2
2023-01-11T20:54:48.6358096Z dist init r=0, world=2
2023-01-11T20:54:48.6358307Z skip: Need at least 2 CUDA devices (1.612s)
2023-01-11T20:54:48.6358563Z   test_materialize_meta_module (__main__.TestFSDPInitialization)
2023-01-11T20:54:48.6359025Z Tests materializing a meta-device module. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13012
2023-01-11T20:54:48.6359397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13013
2023-01-11T20:54:48.6359843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6360157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6360578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6360918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6361246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:54:48.6361695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6362019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6362437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6362759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6363092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:54:48.6363580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6364090Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6364457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:54:48.6364800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:54:48.6365052Z dist init r=1, world=2
2023-01-11T20:54:48.6365219Z dist init r=0, world=2
2023-01-11T20:54:48.6365424Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:54:48.6365695Z   test_nested_fully_shard_shared_state (__main__.TestFSDPInitialization)
2023-01-11T20:54:48.6366090Z Tests that nested applications of ``fully_shard`` share the expected ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13040
2023-01-11T20:54:48.6366514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13041
2023-01-11T20:54:48.6366961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6367284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6367693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6368033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6368361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:54:48.6368823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6369162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6369590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6369930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6370263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:54:48.6370742Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6371486Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6371872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:54:48.6372206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:54:48.6372462Z dist init r=1, world=2
2023-01-11T20:54:48.6372647Z dist init r=0, world=2
2023-01-11T20:54:48.6372854Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:54:48.6373087Z   test_policy (__main__.TestFSDPInitialization)
2023-01-11T20:54:48.6373555Z Tests passing a ``policy`` for pseudo-auto-wrapping. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13068
2023-01-11T20:54:48.6373943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13069
2023-01-11T20:54:48.6374377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6374701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6375127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6375469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6375790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:54:48.6376254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6376573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6377036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6377379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6377715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:54:48.6378208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6378712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6379175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:54:48.6379627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:54:48.6379891Z dist init r=0, world=2
2023-01-11T20:54:48.6380060Z dist init r=1, world=2
2023-01-11T20:54:48.6380267Z skip: Need at least 2 CUDA devices (1.612s)
2023-01-11T20:54:48.6380532Z   test_sync_module_states (__main__.TestFSDPInitialization)
2023-01-11T20:54:48.6380874Z Tests passing ``sync_module_states=True``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13096
2023-01-11T20:54:48.6381242Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13097
2023-01-11T20:54:48.6381697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6382010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6382481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6382824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6383153Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:54:48.6383605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6383926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6384348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6384685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6385003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:54:48.6385495Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6386015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6386466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:54:48.6386811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:54:48.6387055Z dist init r=1, world=2
2023-01-11T20:54:48.6387234Z dist init r=0, world=2
2023-01-11T20:54:48.6387442Z skip: Need at least 2 CUDA devices (1.624s)
2023-01-11T20:54:48.6387708Z   test_state_dict_save_load_flow (__main__.TestFSDPModelCheckpointing)
2023-01-11T20:54:48.6388135Z E2E test of save + load with rank0_only + CPU offload for TransformerWithSharedParams ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13124
2023-01-11T20:54:48.6388555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13125
2023-01-11T20:54:48.6389006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6389321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6389745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6390086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6390418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:54:48.6390867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6391188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6391609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6392001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6392329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:54:48.6392822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6393339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6393710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:54:48.6394062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:54:48.6394320Z dist init r=1, world=2
2023-01-11T20:54:48.6394500Z dist init r=0, world=2
2023-01-11T20:54:48.6394693Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:54:48.6395013Z   test_state_dict_save_load_root_fully_shard (__main__.TestFSDPModelCheckpointing)
2023-01-11T20:54:48.6395422Z Tests that the full state dict saved from a module with ``fully_shard`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13152
2023-01-11T20:54:48.6395796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13153
2023-01-11T20:54:48.6396242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6396565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6396989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6397315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6397652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:54:48.6398112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6398425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6398848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6399188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6399518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:54:48.6399993Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6400504Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6400883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:54:48.6401230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:54:48.6401476Z dist init r=0, world=2
2023-01-11T20:54:48.6401653Z dist init r=1, world=2
2023-01-11T20:54:48.6401858Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:54:48.6402140Z   test_state_dict_save_load_submodule_fully_shard (__main__.TestFSDPModelCheckpointing)
2023-01-11T20:54:48.6402553Z Tests that the full state dict saved from a module with ``fully_shard`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13180
2023-01-11T20:54:48.6402939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13181
2023-01-11T20:54:48.6403379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6403703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6404126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6404500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6404817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:54:48.6405285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6405607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6406025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6406347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6406673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:54:48.6407162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6407693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6408077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:54:48.6408422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:54:48.6408674Z dist init r=1, world=2
2023-01-11T20:54:48.6408839Z dist init r=0, world=2
2023-01-11T20:54:48.6409044Z skip: Need at least 2 CUDA devices (1.612s)
2023-01-11T20:54:48.6409363Z   test_optim_state_dict_save_load (__main__.TestFSDPOptimStateDict) ... skip: The test currently fails on CI. (0.001s)
2023-01-11T20:54:48.6409752Z   test_optim_state_dict_submodule_fully_shard (__main__.TestFSDPOptimStateDict) ... skip: The test currently fails on CI. (0.001s)
2023-01-11T20:54:48.6410067Z   test_training (__main__.TestFSDPRuntime)
2023-01-11T20:54:48.6410408Z Tests training (forward, backward, optimizer). ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13208
2023-01-11T20:54:48.6410787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13209
2023-01-11T20:54:48.6411412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6411741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6412166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6412510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6412830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:54:48.6413295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6413619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6414029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6414374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6414705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:54:48.6415191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6415692Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6416072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:54:48.6416414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:54:48.6416655Z dist init r=1, world=2
2023-01-11T20:54:48.6416905Z dist init r=0, world=2
2023-01-11T20:54:48.6417206Z skip: Need at least 2 CUDA devices (1.612s)
2023-01-11T20:54:48.6417460Z   test_unshard_reshard_order (__main__.TestFSDPRuntime)
2023-01-11T20:54:48.6417822Z Tests that the unshard/reshard order matches between ``fully_shard`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13236
2023-01-11T20:54:48.6418217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13237
2023-01-11T20:54:48.6418674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6418986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6419411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6419755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6420129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:54:48.6420589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6420910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6421330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6421654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6421980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:54:48.6422468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6422983Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6423361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:54:48.6423706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:54:48.6423961Z dist init r=1, world=2
2023-01-11T20:54:48.6424141Z dist init r=0, world=2
2023-01-11T20:54:48.6424334Z skip: Need at least 2 CUDA devices (1.612s)
2023-01-11T20:54:48.6424683Z   test_float16_on_one_submodule (__main__.TestMixedPrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13264
2023-01-11T20:54:48.6425071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13265
2023-01-11T20:54:48.6425508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6425831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6426258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6426598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6426917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:54:48.6427377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:54:48.6427697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:54:48.6428109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:54:48.6428443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:54:48.6428774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:54:48.6429266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6429824Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:54:48.6430205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:54:48.6430543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:54:48.6430803Z dist init r=0, world=2
2023-01-11T20:54:48.6430967Z dist init r=1, world=2
2023-01-11T20:54:48.6431173Z skip: Need at least 2 CUDA devices (1.612s)
2023-01-11T20:54:48.6431310Z 
2023-01-11T20:54:48.6431510Z ----------------------------------------------------------------------
2023-01-11T20:54:48.6431740Z Ran 14 tests in 19.615s
2023-01-11T20:54:48.6431852Z 
2023-01-11T20:54:48.6431926Z OK (skipped=14)
2023-01-11T20:54:48.6432034Z 
2023-01-11T20:54:48.6432119Z Generating XML reports...
2023-01-11T20:54:48.6432606Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPInitialization-20230111205428.xml
2023-01-11T20:54:48.6433238Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPModelCheckpointing-20230111205428.xml
2023-01-11T20:54:48.6433859Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPOptimStateDict-20230111205428.xml
2023-01-11T20:54:48.6434448Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPRuntime-20230111205428.xml
2023-01-11T20:54:48.6435004Z Generated XML report: test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestMixedPrecision-20230111205428.xml
2023-01-11T20:54:48.6435258Z 
2023-01-11T20:54:48.6435578Z ##[endgroup]
2023-01-11T20:54:48.6436047Z FINISHED PRINTING LOG FILE of distributed/_composable/test_fully_shard (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_fully_shard_njr6pcid)
2023-01-11T20:54:48.6436315Z 
2023-01-11T20:54:48.6436516Z Running distributed/fsdp/test_fsdp_grad_acc ... [2023-01-11 20:54:48.634362]
2023-01-11T20:54:48.6437019Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_grad_acc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:54:48.634728]
2023-01-11T20:55:09.8233038Z 
2023-01-11T20:55:09.8242140Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_grad_acc
2023-01-11T20:55:09.8243933Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_grad_acc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_grad_acc_kajrqa6a)
2023-01-11T20:55:09.8244843Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc
2023-01-11T20:55:09.8245168Z 
2023-01-11T20:55:09.8245431Z Running tests...
2023-01-11T20:55:09.8245973Z ----------------------------------------------------------------------
2023-01-11T20:55:09.8246765Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_sharding_strategy_ShardingStrategy_FULL_SHARD_use_orig_params_False (__main__.TestGradAcc)
2023-01-11T20:55:09.8247582Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13303
2023-01-11T20:55:09.8248157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13304
2023-01-11T20:55:09.8248879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8249264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8249753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8250097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8250487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:55:09.8251462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8251795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8252289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8252621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8253023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:55:09.8253554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8254108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8254622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:55:09.8254978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:55:09.8255306Z dist init r=0, world=2
2023-01-11T20:55:09.8255478Z dist init r=1, world=2
2023-01-11T20:55:09.8255684Z skip: Need at least 2 CUDA devices (1.861s)
2023-01-11T20:55:09.8256159Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_sharding_strategy_ShardingStrategy_FULL_SHARD_use_orig_params_True (__main__.TestGradAcc)
2023-01-11T20:55:09.8256639Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13331
2023-01-11T20:55:09.8257135Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13332
2023-01-11T20:55:09.8257665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8258000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8258483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8258832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8259234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:55:09.8259701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8260074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8260501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8260907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8261236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:55:09.8261792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8262364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8262752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:55:09.8263237Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:55:09.8263495Z dist init r=0, world=2
2023-01-11T20:55:09.8263661Z dist init r=1, world=2
2023-01-11T20:55:09.8263928Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:55:09.8264334Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_sharding_strategy_ShardingStrategy_NO_SHARD_use_orig_params_False (__main__.TestGradAcc)
2023-01-11T20:55:09.8264863Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13359
2023-01-11T20:55:09.8265288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13360
2023-01-11T20:55:09.8265811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8266179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8266880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8267420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8267922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:55:09.8268629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8269061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8269785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8270346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8270884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:55:09.8271671Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8272824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8273441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:55:09.8274011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:55:09.8274434Z dist init r=0, world=2
2023-01-11T20:55:09.8274718Z dist init r=1, world=2
2023-01-11T20:55:09.8275019Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:55:09.8275595Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_sharding_strategy_ShardingStrategy_NO_SHARD_use_orig_params_True (__main__.TestGradAcc)
2023-01-11T20:55:09.8276287Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13387
2023-01-11T20:55:09.8276814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13388
2023-01-11T20:55:09.8277522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8278015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8278699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8279252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8279792Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:55:09.8280598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8281126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8281816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8282347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8282879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:55:09.8283681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8284529Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8285262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:55:09.8285788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:55:09.8286191Z dist init r=1, world=2
2023-01-11T20:55:09.8286458Z dist init r=0, world=2
2023-01-11T20:55:09.8286772Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:55:09.8287416Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_use_orig_params_False (__main__.TestGradAcc)
2023-01-11T20:55:09.8288202Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13415
2023-01-11T20:55:09.8288732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13416
2023-01-11T20:55:09.8289531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8290061Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8290752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8291421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8291912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:55:09.8292595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8293054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8293643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8294137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8294634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:55:09.8295408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8296239Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8296841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:55:09.8297484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:55:09.8297897Z dist init r=1, world=2
2023-01-11T20:55:09.8298178Z dist init r=0, world=2
2023-01-11T20:55:09.8298524Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:55:09.8299260Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_use_orig_params_True (__main__.TestGradAcc)
2023-01-11T20:55:09.8300116Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13443
2023-01-11T20:55:09.8300775Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13444
2023-01-11T20:55:09.8301548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8302087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8302807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8303385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8303938Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:55:09.8304777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8305497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8306281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8306890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8307439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:55:09.8308324Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8309214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8309898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:55:09.8310638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:55:09.8311091Z dist init r=1, world=2
2023-01-11T20:55:09.8311387Z dist init r=0, world=2
2023-01-11T20:55:09.8311751Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:55:09.8312449Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_sharding_strategy_ShardingStrategy_FULL_SHARD_use_orig_params_False (__main__.TestGradAcc)
2023-01-11T20:55:09.8313283Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13471
2023-01-11T20:55:09.8313952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13472
2023-01-11T20:55:09.8314793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8315386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8316181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8316789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8317356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:55:09.8318180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8318735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8319517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8320129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8320722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:55:09.8321605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8322562Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8323248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:55:09.8323845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:55:09.8324308Z dist init r=0, world=2
2023-01-11T20:55:09.8324614Z dist init r=1, world=2
2023-01-11T20:55:09.8324965Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:55:09.8325686Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_sharding_strategy_ShardingStrategy_FULL_SHARD_use_orig_params_True (__main__.TestGradAcc)
2023-01-11T20:55:09.8326557Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13499
2023-01-11T20:55:09.8327315Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13500
2023-01-11T20:55:09.8328139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8328720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8329489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8330101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8330688Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:55:09.8331704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8332272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8333146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8333780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8334389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:55:09.8335293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8336204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8336908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:55:09.8337607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:55:09.8338062Z dist init r=0, world=2
2023-01-11T20:55:09.8338357Z dist init r=1, world=2
2023-01-11T20:55:09.8338721Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:55:09.8339429Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_sharding_strategy_ShardingStrategy_NO_SHARD_use_orig_params_False (__main__.TestGradAcc)
2023-01-11T20:55:09.8340223Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13527
2023-01-11T20:55:09.8340887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13528
2023-01-11T20:55:09.8341718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8342293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8343055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8343652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8344190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:55:09.8344973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8345578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8346361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8346967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8347524Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:55:09.8348398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8349346Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8350221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:55:09.8350832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:55:09.8351286Z dist init r=1, world=2
2023-01-11T20:55:09.8351586Z dist init r=0, world=2
2023-01-11T20:55:09.8351909Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:55:09.8352605Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_sharding_strategy_ShardingStrategy_NO_SHARD_use_orig_params_True (__main__.TestGradAcc)
2023-01-11T20:55:09.8353467Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13555
2023-01-11T20:55:09.8354111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13556
2023-01-11T20:55:09.8354997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8355582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8356362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8356954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8357557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:55:09.8358423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8358987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8359745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8360362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8360978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:55:09.8361929Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8362881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8363591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:55:09.8364217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:55:09.8364651Z dist init r=0, world=2
2023-01-11T20:55:09.8364966Z dist init r=1, world=2
2023-01-11T20:55:09.8365335Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:55:09.8366053Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_use_orig_params_False (__main__.TestGradAcc)
2023-01-11T20:55:09.8366907Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13583
2023-01-11T20:55:09.8367561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13584
2023-01-11T20:55:09.8368413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8368988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8369751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8370359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8371194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:55:09.8372025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8372748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8373521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8374124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8374689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:55:09.8375614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8376563Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8377311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:55:09.8378064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:55:09.8378512Z dist init r=0, world=2
2023-01-11T20:55:09.8378811Z dist init r=1, world=2
2023-01-11T20:55:09.8379154Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:55:09.8379857Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_use_orig_params_True (__main__.TestGradAcc)
2023-01-11T20:55:09.8380706Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13611
2023-01-11T20:55:09.8381342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13612
2023-01-11T20:55:09.8382180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8382755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8383536Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8384160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8384732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:55:09.8385586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:09.8386170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:09.8386895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:09.8387492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:09.8388079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:55:09.8388972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8389939Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:55:09.8390627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:55:09.8391234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:55:09.8391664Z dist init r=0, world=2
2023-01-11T20:55:09.8391984Z dist init r=1, world=2
2023-01-11T20:55:09.8392339Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:55:09.8392582Z 
2023-01-11T20:55:09.8392938Z ----------------------------------------------------------------------
2023-01-11T20:55:09.8393371Z Ran 12 tests in 19.604s
2023-01-11T20:55:09.8393568Z 
2023-01-11T20:55:09.8393693Z OK (skipped=12)
2023-01-11T20:55:09.8393886Z 
2023-01-11T20:55:09.8394034Z Generating XML reports...
2023-01-11T20:55:09.8394824Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20230111205449.xml
2023-01-11T20:55:09.8395377Z 
2023-01-11T20:55:09.8395844Z ##[endgroup]
2023-01-11T20:55:09.8396675Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_grad_acc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_grad_acc_kajrqa6a)
2023-01-11T20:55:09.8397135Z 
2023-01-11T20:55:09.8397483Z Running distributed/_tensor/test_view_ops ... [2023-01-11 20:55:09.823961]
2023-01-11T20:55:09.8398404Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_view_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:55:09.824355]
2023-01-11T20:55:37.7743233Z 
2023-01-11T20:55:37.7744095Z Expand the folded group to see the log file of distributed/_tensor/test_view_ops
2023-01-11T20:55:37.7745389Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_view_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_view_ops_eb3e10os)
2023-01-11T20:55:37.7746509Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_view_ops
2023-01-11T20:55:37.7746987Z 
2023-01-11T20:55:37.7747128Z Running tests...
2023-01-11T20:55:37.7747678Z ----------------------------------------------------------------------
2023-01-11T20:55:37.7748040Z   test_view_groups (__main__.TestViewOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13650
2023-01-11T20:55:37.7748546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13651
2023-01-11T20:55:37.7748999Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13652
2023-01-11T20:55:37.7749312Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13653
2023-01-11T20:55:37.7749695Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 13654
2023-01-11T20:55:37.7750087Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 13655
2023-01-11T20:55:37.7750567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:37.7750888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:37.7751320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:37.7751667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:37.7751997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:55:37.7752674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:37.7753056Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:37.7753659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:37.7754286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:37.7754619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:55:37.7755077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:37.7755402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:37.7755815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:37.7756157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:37.7756476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:55:37.7756917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:37.7757240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:37.7757664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:37.7758145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:37.7758451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:55:37.7758907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:37.7759232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:37.7759640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:37.7759979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:37.7760299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:55:37.7760794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:37.7761107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:37.7761530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:37.7761868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:37.7762187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:55:37.7762425Z ok (2.579s)
2023-01-11T20:55:37.7762718Z   test_view_ops (__main__.TestViewOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13716
2023-01-11T20:55:37.7763081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13717
2023-01-11T20:55:37.7763394Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13718
2023-01-11T20:55:37.7763715Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13719
2023-01-11T20:55:37.7764041Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 13720
2023-01-11T20:55:37.7764350Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 13721
2023-01-11T20:55:37.7764796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:37.7765118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:37.7765541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:37.7765869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:37.7766186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T20:55:37.7766638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:37.7766962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:37.7767375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:37.7767713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:37.7768034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:55:37.7768479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:37.7768799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:37.7769227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:37.7769566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:37.7769872Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:55:37.7770368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:37.7770690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:37.7771308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:37.7771648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:37.7771967Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:55:37.7772420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:37.7772729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:37.7773149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:37.7773553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:37.7773874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T20:55:37.7774318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:55:37.7774640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:55:37.7775060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:55:37.7775386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:55:37.7775703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:55:37.7776067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:55:37.7776433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:55:37.7776782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:55:37.7777217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:55:37.7777575Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 5
2023-01-11T20:55:37.7777926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 4
2023-01-11T20:55:37.7778406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T20:55:37.7778923Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T20:55:37.7779437Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T20:55:37.7779934Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T20:55:37.7780324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T20:55:37.7780683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 5
2023-01-11T20:55:37.7781038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T20:55:37.7781510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T20:55:37.7782020Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 6 nodes.
2023-01-11T20:55:37.7782409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T20:55:37.7782771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T20:55:37.7783174Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 4
2023-01-11T20:55:37.7783661Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T20:55:37.7784171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T20:55:37.7784680Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T20:55:37.7785053Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T20:55:37.7785408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 4
2023-01-11T20:55:37.7785764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T20:55:37.7786279Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T20:55:37.7786774Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T20:55:37.7787280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 6 nodes.
2023-01-11T20:55:37.7787669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T20:55:37.7788013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 5
2023-01-11T20:55:37.7788377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T20:55:37.7788858Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:3 with 6 nodes.
2023-01-11T20:55:37.7789370Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 6 nodes.
2023-01-11T20:55:37.7789745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T20:55:37.7790104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 5
2023-01-11T20:55:37.7790586Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 6 nodes.
2023-01-11T20:55:37.7791095Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:3 with 6 nodes.
2023-01-11T20:55:37.7791590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 6 nodes.
2023-01-11T20:55:37.7809338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T20:55:37.7809927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 4
2023-01-11T20:55:37.7810505Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 6 nodes.
2023-01-11T20:55:37.7811012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T20:55:37.7811376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T20:55:37.7811860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 6 nodes.
2023-01-11T20:55:37.7812373Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 6 nodes.
2023-01-11T20:55:37.7812765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T20:55:37.7813131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T20:55:37.7813755Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 6 nodes.
2023-01-11T20:55:37.7814269Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:4 with 6 nodes.
2023-01-11T20:55:37.7814666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 4
2023-01-11T20:55:37.7815151Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 6 nodes.
2023-01-11T20:55:37.7815643Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:4 with 6 nodes.
2023-01-11T20:55:37.7816034Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 5
2023-01-11T20:55:37.7816395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T20:55:37.7816802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T20:55:37.7817363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 6 nodes.
2023-01-11T20:55:37.7817878Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 6 nodes.
2023-01-11T20:55:37.7818444Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 6 nodes.
2023-01-11T20:55:37.7818819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1
2023-01-11T20:55:37.7819180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 2
2023-01-11T20:55:37.7819540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 3
2023-01-11T20:55:37.7820029Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:5 with 6 nodes.
2023-01-11T20:55:37.7820527Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:5 with 6 nodes.
2023-01-11T20:55:37.7821030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 6 nodes.
2023-01-11T20:55:37.7821420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0
2023-01-11T20:55:37.7821778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 4
2023-01-11T20:55:37.7822124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 5
2023-01-11T20:55:37.7822603Z INFO:torch.distributed.distributed_c10d:Rank 5: Completed store-based barrier for key:store_based_barrier_key:6 with 6 nodes.
2023-01-11T20:55:37.7823110Z INFO:torch.distributed.distributed_c10d:Rank 4: Completed store-based barrier for key:store_based_barrier_key:6 with 6 nodes.
2023-01-11T20:55:37.7823621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 6 nodes.
2023-01-11T20:55:37.7824108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 6 nodes.
2023-01-11T20:55:37.7824612Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:6 with 6 nodes.
2023-01-11T20:55:37.7825113Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:6 with 6 nodes.
2023-01-11T20:55:37.7825396Z ok (23.818s)
2023-01-11T20:55:37.7825488Z 
2023-01-11T20:55:37.7825688Z ----------------------------------------------------------------------
2023-01-11T20:55:37.7825927Z Ran 2 tests in 26.397s
2023-01-11T20:55:37.7826041Z 
2023-01-11T20:55:37.7826102Z OK
2023-01-11T20:55:37.7826239Z 
2023-01-11T20:55:37.7826313Z Generating XML reports...
2023-01-11T20:55:37.7826746Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_view_ops/TEST-TestViewOps-20230111205511.xml
2023-01-11T20:55:37.7826985Z 
2023-01-11T20:55:37.7827305Z ##[endgroup]
2023-01-11T20:55:37.7827734Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_view_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_view_ops_eb3e10os)
2023-01-11T20:55:37.7827982Z 
2023-01-11T20:55:37.7828207Z Running distributed/fsdp/test_fsdp_checkpoint ... [2023-01-11 20:55:37.774760]
2023-01-11T20:55:37.7828729Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:55:37.775165]
2023-01-11T20:56:09.8452354Z 
2023-01-11T20:56:09.8452835Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_checkpoint
2023-01-11T20:56:09.8454198Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_checkpoint_d2jsjptj)
2023-01-11T20:56:09.8455165Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint
2023-01-11T20:56:09.8489356Z 
2023-01-11T20:56:09.8489629Z Running tests...
2023-01-11T20:56:09.8490138Z ----------------------------------------------------------------------
2023-01-11T20:56:09.8491037Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13937
2023-01-11T20:56:09.8491790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13938
2023-01-11T20:56:09.8492130Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13939
2023-01-11T20:56:09.8492525Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13940
2023-01-11T20:56:09.8493149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8493481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8493922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8494271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8494812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:09.8495471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8495803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8496224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8496560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8496895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:09.8497447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8497772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8498181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8498522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8498851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:56:09.8499334Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8499855Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8500512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8500839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8501251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8501594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8501928Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:56:09.8502420Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8502924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8503376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:09.8503727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:56:09.8504052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:09.8504390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:56:09.8504644Z dist init r=1, world=4
2023-01-11T20:56:09.8504825Z dist init r=0, world=4
2023-01-11T20:56:09.8504987Z dist init r=3, world=4
2023-01-11T20:56:09.8505162Z dist init r=2, world=4
2023-01-11T20:56:09.8505367Z skip: Need at least 2 CUDA devices (2.185s)
2023-01-11T20:56:09.8505809Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13993
2023-01-11T20:56:09.8506306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13994
2023-01-11T20:56:09.8506631Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 13995
2023-01-11T20:56:09.8506951Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 13996
2023-01-11T20:56:09.8507390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8507718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8508138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8508469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8508801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:09.8509268Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8509590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8509996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8510336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8510663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:09.8511110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8511432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8511849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8512189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8512551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:56:09.8513013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8513337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8513754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8514075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8514404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:56:09.8514891Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8515432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8515946Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8516458Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8516844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:56:09.8517179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:56:09.8517514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:09.8517849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:09.8518103Z dist init r=2, world=4
2023-01-11T20:56:09.8518270Z dist init r=3, world=4
2023-01-11T20:56:09.8518445Z dist init r=1, world=4
2023-01-11T20:56:09.8518619Z dist init r=0, world=4
2023-01-11T20:56:09.8518813Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T20:56:09.8519265Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14049
2023-01-11T20:56:09.8519753Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14050
2023-01-11T20:56:09.8520065Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14051
2023-01-11T20:56:09.8520387Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14052
2023-01-11T20:56:09.8520832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8521158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8521572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8521914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8522243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:09.8522706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8523016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8523432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8523766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8524081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:56:09.8524539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8524903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8525322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8525642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8525969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:09.8526462Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8526953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8527261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8527683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8528058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8528376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:56:09.8528861Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8529377Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8529891Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8530260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:09.8530606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:56:09.8531131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:09.8531478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:56:09.8531721Z dist init r=0, world=4
2023-01-11T20:56:09.8531901Z dist init r=1, world=4
2023-01-11T20:56:09.8532075Z dist init r=3, world=4
2023-01-11T20:56:09.8532237Z dist init r=2, world=4
2023-01-11T20:56:09.8532441Z skip: Need at least 2 CUDA devices (1.721s)
2023-01-11T20:56:09.8532893Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14105
2023-01-11T20:56:09.8533366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14106
2023-01-11T20:56:09.8533693Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14107
2023-01-11T20:56:09.8534010Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14108
2023-01-11T20:56:09.8534464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8534795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8535223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8535551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8535883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:09.8536345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8536670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8537079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8537488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8537907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:09.8538371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8538680Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8539334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8539676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8539993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:56:09.8540485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8541026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8541351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8541761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8542099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8542426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:56:09.8542914Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8543413Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8543926Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8544313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:09.8544661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:09.8544987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:56:09.8545318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:56:09.8545575Z dist init r=1, world=4
2023-01-11T20:56:09.8545739Z dist init r=2, world=4
2023-01-11T20:56:09.8545914Z dist init r=0, world=4
2023-01-11T20:56:09.8546091Z dist init r=3, world=4
2023-01-11T20:56:09.8546280Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T20:56:09.8546735Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14161
2023-01-11T20:56:09.8547220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14162
2023-01-11T20:56:09.8547543Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14163
2023-01-11T20:56:09.8547848Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14164
2023-01-11T20:56:09.8548292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8548616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8549027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8549365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8549694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:09.8550157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8550504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8550924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8551262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8551588Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:09.8552066Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8552554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8552878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8553317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8553659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8553989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:56:09.8554452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8554760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8555179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8555515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8555842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:56:09.8556316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8556834Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8557345Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8557713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:09.8558057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:56:09.8558392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:09.8558728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:56:09.8558969Z dist init r=0, world=4
2023-01-11T20:56:09.8559145Z dist init r=1, world=4
2023-01-11T20:56:09.8559320Z dist init r=2, world=4
2023-01-11T20:56:09.8559481Z dist init r=3, world=4
2023-01-11T20:56:09.8559686Z skip: Need at least 2 CUDA devices (1.721s)
2023-01-11T20:56:09.8560138Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14217
2023-01-11T20:56:09.8560622Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14218
2023-01-11T20:56:09.8560937Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14219
2023-01-11T20:56:09.8561258Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14220
2023-01-11T20:56:09.8561711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8562024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8562452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8562829Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8563162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:09.8563615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8563940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8564362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8564701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8565017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:09.8565483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8565843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8566253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8566594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8566925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:56:09.8567389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8567697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8568119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8568458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8568779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:56:09.8569272Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8569788Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8570298Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8570794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8571337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:09.8571680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:56:09.8572024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:09.8572353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:56:09.8572610Z dist init r=2, world=4
2023-01-11T20:56:09.8572791Z dist init r=1, world=4
2023-01-11T20:56:09.8572957Z dist init r=0, world=4
2023-01-11T20:56:09.8573137Z dist init r=3, world=4
2023-01-11T20:56:09.8573343Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T20:56:09.8573784Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14273
2023-01-11T20:56:09.8574264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14274
2023-01-11T20:56:09.8574585Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14275
2023-01-11T20:56:09.8574906Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14276
2023-01-11T20:56:09.8575415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8575742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8576164Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8576506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8576821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:09.8577357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8577682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8578090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8578490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8578826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:09.8579286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8579590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8580006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8580346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8580675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:56:09.8581125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8581448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8581872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8582197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8582526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:56:09.8583008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8583522Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8584016Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8584523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8584907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:56:09.8585250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:09.8585576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:09.8585911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:56:09.8586163Z dist init r=2, world=4
2023-01-11T20:56:09.8586328Z dist init r=1, world=4
2023-01-11T20:56:09.8586501Z dist init r=0, world=4
2023-01-11T20:56:09.8586673Z dist init r=3, world=4
2023-01-11T20:56:09.8586862Z skip: Need at least 2 CUDA devices (1.721s)
2023-01-11T20:56:09.8587312Z   test_basic_checkpoint_end_to_end_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14329
2023-01-11T20:56:09.8587831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14330
2023-01-11T20:56:09.8588158Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14331
2023-01-11T20:56:09.8588466Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14332
2023-01-11T20:56:09.8588912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8589233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8589656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8589982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8590314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:09.8590815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8591128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8591548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8591889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8592216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:56:09.8592689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8593180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8593501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8593923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8594250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8594579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:09.8595038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8595347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8595764Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8596106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8596431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:56:09.8596902Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8597418Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8597927Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8598304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:56:09.8598631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:09.8598963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:56:09.8599301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:09.8599540Z dist init r=3, world=4
2023-01-11T20:56:09.8599718Z dist init r=2, world=4
2023-01-11T20:56:09.8599894Z dist init r=1, world=4
2023-01-11T20:56:09.8600056Z dist init r=0, world=4
2023-01-11T20:56:09.8600299Z skip: Need at least 2 CUDA devices (1.721s)
2023-01-11T20:56:09.8600758Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14385
2023-01-11T20:56:09.8601240Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14386
2023-01-11T20:56:09.8601551Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14387
2023-01-11T20:56:09.8601866Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14388
2023-01-11T20:56:09.8602314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8602628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8603048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8603425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8603757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:09.8604209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8604533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8604955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8605291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8605611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:09.8606072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8606397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8606804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8607138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8607466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:56:09.8607957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8608435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8608759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8609175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8609512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8609826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:56:09.8610310Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8610961Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8611498Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8611883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:09.8612229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:09.8612568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:56:09.8612960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:56:09.8613217Z dist init r=2, world=4
2023-01-11T20:56:09.8613399Z dist init r=1, world=4
2023-01-11T20:56:09.8613562Z dist init r=0, world=4
2023-01-11T20:56:09.8613738Z dist init r=3, world=4
2023-01-11T20:56:09.8613944Z skip: Need at least 2 CUDA devices (1.721s)
2023-01-11T20:56:09.8614401Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14441
2023-01-11T20:56:09.8614873Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14442
2023-01-11T20:56:09.8615200Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14443
2023-01-11T20:56:09.8615524Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14444
2023-01-11T20:56:09.8616001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8616331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8616757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8617102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8617478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:09.8617943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8618271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8618693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8619022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8619354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:09.8619817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8620130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8620552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8620891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8621219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:56:09.8621696Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8622193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8622519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8622931Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8623273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8623605Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:56:09.8624097Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8624598Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8625111Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8625494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:56:09.8625880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:09.8626207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:09.8626538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:56:09.8626795Z dist init r=1, world=4
2023-01-11T20:56:09.8626960Z dist init r=2, world=4
2023-01-11T20:56:09.8627134Z dist init r=3, world=4
2023-01-11T20:56:09.8627307Z dist init r=0, world=4
2023-01-11T20:56:09.8627496Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T20:56:09.8627954Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14497
2023-01-11T20:56:09.8628474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14498
2023-01-11T20:56:09.8628807Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14499
2023-01-11T20:56:09.8629116Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14500
2023-01-11T20:56:09.8629559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8629886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8630309Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8630638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8630966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:09.8631428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8631739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8632158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8632495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8632826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:09.8633270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8633589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8634010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8634350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8634667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:56:09.8635130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8635447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8635849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8636184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8636511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:56:09.8636999Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8637496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8638042Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8638549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8638928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:09.8639257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:09.8639593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:56:09.8639929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:56:09.8640170Z dist init r=1, world=4
2023-01-11T20:56:09.8640347Z dist init r=0, world=4
2023-01-11T20:56:09.8640520Z dist init r=2, world=4
2023-01-11T20:56:09.8640681Z dist init r=3, world=4
2023-01-11T20:56:09.8640921Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T20:56:09.8641374Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=False)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14553
2023-01-11T20:56:09.8641852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14554
2023-01-11T20:56:09.8642165Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14555
2023-01-11T20:56:09.8642482Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14556
2023-01-11T20:56:09.8642930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8643253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8643661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8644008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8644342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:09.8644796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8645117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8645535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8645871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8646187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:09.8646652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8646978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8647388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8647724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8648049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:56:09.8648536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8649014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8649334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8649754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8650095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8650449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:56:09.8651086Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8651605Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8652116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8652481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:56:09.8652827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:09.8653198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:09.8653580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:56:09.8653837Z dist init r=0, world=4
2023-01-11T20:56:09.8654019Z dist init r=2, world=4
2023-01-11T20:56:09.8654183Z dist init r=1, world=4
2023-01-11T20:56:09.8654359Z dist init r=3, world=4
2023-01-11T20:56:09.8654564Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T20:56:09.8655022Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14609
2023-01-11T20:56:09.8655490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14610
2023-01-11T20:56:09.8655818Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14611
2023-01-11T20:56:09.8656136Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14612
2023-01-11T20:56:09.8656575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8656904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8657397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8657741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8658062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:09.8658532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8658856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8659279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8659607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8659944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:09.8660404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8660713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8661134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8661479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8661806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:56:09.8662253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8662578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8663068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8663405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8663720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:56:09.8664220Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8664737Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8665237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8665749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8666179Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:56:09.8666530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:09.8666857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:09.8667194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:56:09.8667451Z dist init r=2, world=4
2023-01-11T20:56:09.8667617Z dist init r=1, world=4
2023-01-11T20:56:09.8667794Z dist init r=3, world=4
2023-01-11T20:56:09.8667970Z dist init r=0, world=4
2023-01-11T20:56:09.8668159Z skip: Need at least 2 CUDA devices (1.721s)
2023-01-11T20:56:09.8668620Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_False_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14665
2023-01-11T20:56:09.8669105Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14666
2023-01-11T20:56:09.8669439Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14667
2023-01-11T20:56:09.8669755Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14668
2023-01-11T20:56:09.8670202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8670528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8670955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8671282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8671615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:09.8672079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8672408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8672821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8673162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8673492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:09.8673944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8674270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8674689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8675030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8675351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:56:09.8675857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8676181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8676591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8676929Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8677259Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:56:09.8677750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8678255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8678802Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8679320Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8679703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:09.8680038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:09.8680374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:56:09.8680716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:56:09.8680963Z dist init r=1, world=4
2023-01-11T20:56:09.8681146Z dist init r=0, world=4
2023-01-11T20:56:09.8681322Z dist init r=3, world=4
2023-01-11T20:56:09.8681482Z dist init r=2, world=4
2023-01-11T20:56:09.8681688Z skip: Need at least 2 CUDA devices (1.822s)
2023-01-11T20:56:09.8682140Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14721
2023-01-11T20:56:09.8682624Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14722
2023-01-11T20:56:09.8682935Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14723
2023-01-11T20:56:09.8683249Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14724
2023-01-11T20:56:09.8683694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8684019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8684429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8684778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8685109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:09.8685563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8685883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8686304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8686644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8686960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:09.8687425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8687754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8688213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8688537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8688866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:56:09.8689354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8689831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8690152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8690571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8691077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8691463Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:56:09.8691967Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8692486Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8693001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8693373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:09.8693719Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:56:09.8694065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:56:09.8694393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:09.8694657Z dist init r=2, world=4
2023-01-11T20:56:09.8694838Z dist init r=0, world=4
2023-01-11T20:56:09.8695018Z dist init r=3, world=4
2023-01-11T20:56:09.8695180Z dist init r=1, world=4
2023-01-11T20:56:09.8695382Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T20:56:09.8695832Z   test_checkpoint_fsdp_wrapping_cpu_offload_CPUOffload(offload_params=True)_offload_activations_True_use_orig_params_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14777
2023-01-11T20:56:09.8696304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14778
2023-01-11T20:56:09.8696630Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14779
2023-01-11T20:56:09.8696953Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14780
2023-01-11T20:56:09.8697482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8697806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8698237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8698580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8698901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:09.8699365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8699691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8700113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8700436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8700828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:09.8701288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8701599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8702018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8702362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8702689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:56:09.8703139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8703463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8703920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8704258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8704572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:56:09.8705060Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8705573Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8706071Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8706580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8706963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:56:09.8707313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:09.8707640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:09.8707970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:56:09.8708225Z dist init r=3, world=4
2023-01-11T20:56:09.8708403Z dist init r=0, world=4
2023-01-11T20:56:09.8708567Z dist init r=1, world=4
2023-01-11T20:56:09.8708742Z dist init r=2, world=4
2023-01-11T20:56:09.8708943Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T20:56:09.8709330Z   test_checkpoint_submodule_use_reentrant_False (__main__.TestFSDPCheckpointSubmodule) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14833
2023-01-11T20:56:09.8709751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14834
2023-01-11T20:56:09.8710075Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14835
2023-01-11T20:56:09.8710386Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14836
2023-01-11T20:56:09.8710833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8711159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8711583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8711913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8712242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:09.8712708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8713016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8713481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8713819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8714148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:09.8714594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8714914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8715334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8715669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8715981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:56:09.8716477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:09.8716798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:09.8717203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:09.8717535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:09.8717862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:56:09.8718351Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8718853Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8719369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8719884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:09.8720262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:56:09.8720590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:56:09.8720923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:09.8721261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:09.8721504Z dist init r=2, world=4
2023-01-11T20:56:09.8721685Z dist init r=3, world=4
2023-01-11T20:56:09.8721861Z dist init r=1, world=4
2023-01-11T20:56:09.8722024Z dist init r=0, world=4
2023-01-11T20:56:09.8722229Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T20:56:09.8722364Z 
2023-01-11T20:56:09.8722565Z ----------------------------------------------------------------------
2023-01-11T20:56:09.8722807Z Ran 17 tests in 30.419s
2023-01-11T20:56:09.8722908Z 
2023-01-11T20:56:09.8722980Z OK (skipped=17)
2023-01-11T20:56:09.8723088Z 
2023-01-11T20:56:09.8723173Z Generating XML reports...
2023-01-11T20:56:09.8723635Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20230111205539.xml
2023-01-11T20:56:09.8724256Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpointSubmodule-20230111205539.xml
2023-01-11T20:56:09.8724544Z 
2023-01-11T20:56:09.8724916Z ##[endgroup]
2023-01-11T20:56:09.8725382Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_checkpoint_d2jsjptj)
2023-01-11T20:56:09.8725651Z 
2023-01-11T20:56:09.8725865Z Running distributed/fsdp/test_fsdp_use_orig_params ... [2023-01-11 20:56:09.846314]
2023-01-11T20:56:09.8726438Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_use_orig_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:56:09.846659]
2023-01-11T20:56:42.7017964Z 
2023-01-11T20:56:42.7018537Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_use_orig_params
2023-01-11T20:56:42.7019917Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_use_orig_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_use_orig_params_aujsokfw)
2023-01-11T20:56:42.7021257Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params
2023-01-11T20:56:42.7051850Z 
2023-01-11T20:56:42.7052157Z Running tests...
2023-01-11T20:56:42.7052763Z ----------------------------------------------------------------------
2023-01-11T20:56:42.7053273Z   test_named_parameters_in_forward (__main__.TestFSDPUseOrigParamsFQNs)
2023-01-11T20:56:42.7054145Z Tests that calling ``named_parameters()`` during forward returns FQNs ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14900
2023-01-11T20:56:42.7055067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14901
2023-01-11T20:56:42.7055565Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14902
2023-01-11T20:56:42.7056099Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14903
2023-01-11T20:56:42.7056845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7057414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7058114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7058604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7059149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:42.7059854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7060314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7060981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7061619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7062042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:56:42.7062501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7062825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7063249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7063599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7063918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:42.7064383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7064704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7065127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7065451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7065783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:56:42.7066277Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:42.7066926Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:42.7067454Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:42.7067972Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:42.7068359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:56:42.7068694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:56:42.7069040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:42.7069376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:42.7069640Z dist init r=3, world=4
2023-01-11T20:56:42.7069811Z dist init r=1, world=4
2023-01-11T20:56:42.7070045Z dist init r=2, world=4
2023-01-11T20:56:42.7070226Z dist init r=0, world=4
2023-01-11T20:56:42.7070419Z skip: Need at least 2 CUDA devices (2.071s)
2023-01-11T20:56:42.7070696Z   test_param_and_buffer_names (__main__.TestFSDPUseOrigParamsFQNs)
2023-01-11T20:56:42.7071090Z Tests that, for ``use_orig_params=True``, the parameter and buffer ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14956
2023-01-11T20:56:42.7071469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14957
2023-01-11T20:56:42.7071800Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 14958
2023-01-11T20:56:42.7072122Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 14959
2023-01-11T20:56:42.7072577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7072894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7073334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7073683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7074008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:42.7074472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7074797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7075223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7075552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7075888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:56:42.7076358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7076685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7077097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7077444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7077776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:42.7078225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7078546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7078968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7079315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7079670Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:56:42.7080164Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:42.7080686Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:42.7081202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:42.7081700Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:56:42.7082085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:56:42.7082426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:42.7082794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:42.7083136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:56:42.7083387Z dist init r=2, world=4
2023-01-11T20:56:42.7083564Z dist init r=1, world=4
2023-01-11T20:56:42.7083725Z dist init r=0, world=4
2023-01-11T20:56:42.7083900Z dist init r=3, world=4
2023-01-11T20:56:42.7084101Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T20:56:42.7084446Z   test_diff_hyperparams_cpu_offload_sharding_strategy_str_full_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2023-01-11T20:56:42.7084925Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15012
2023-01-11T20:56:42.7085319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15013
2023-01-11T20:56:42.7085772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7086086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7086509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7086850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7087168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:42.7087631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7087952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7088374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7088697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7089032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:42.7089521Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7090021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7090401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:42.7090741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:42.7091197Z dist init r=1, world=2
2023-01-11T20:56:42.7091364Z dist init r=0, world=2
2023-01-11T20:56:42.7091571Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:56:42.7091932Z   test_diff_hyperparams_cpu_offload_sharding_strategy_str_no_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2023-01-11T20:56:42.7092472Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15040
2023-01-11T20:56:42.7092869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15041
2023-01-11T20:56:42.7093328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7093659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7094074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7094416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7094749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:42.7095212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7095568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7095991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7096328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7096644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:42.7097134Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7097717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7098102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:42.7098432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:42.7098690Z dist init r=0, world=2
2023-01-11T20:56:42.7098875Z dist init r=1, world=2
2023-01-11T20:56:42.7099070Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:56:42.7099432Z   test_diff_hyperparams_cpu_offload_sharding_strategy_str_shard_grad_op (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2023-01-11T20:56:42.7099913Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15068
2023-01-11T20:56:42.7100311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15069
2023-01-11T20:56:42.7100747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7101072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7101499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7101842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7102166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:42.7102631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7102952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7103360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7103697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7104028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:42.7104517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7105064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7105444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:42.7105784Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:42.7106040Z dist init r=0, world=2
2023-01-11T20:56:42.7106205Z dist init r=1, world=2
2023-01-11T20:56:42.7106407Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:56:42.7106750Z   test_diff_hyperparams_sharding_strategy_str_full_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2023-01-11T20:56:42.7107200Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15096
2023-01-11T20:56:42.7107592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15097
2023-01-11T20:56:42.7108078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7108392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7108817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7109157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7109488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:42.7109946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7110267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7110695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7111034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7111352Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:42.7111841Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7112355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7112726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:42.7113070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:42.7113322Z dist init r=1, world=2
2023-01-11T20:56:42.7113501Z dist init r=0, world=2
2023-01-11T20:56:42.7113698Z skip: Need at least 2 CUDA devices (1.622s)
2023-01-11T20:56:42.7114040Z   test_diff_hyperparams_sharding_strategy_str_no_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2023-01-11T20:56:42.7114504Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15124
2023-01-11T20:56:42.7114888Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15125
2023-01-11T20:56:42.7115336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7115662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7116084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7116409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7116741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:42.7117207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7117572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7117984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7118321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7118654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:42.7119126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7119636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7120024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:42.7120371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:42.7120655Z dist init r=1, world=2
2023-01-11T20:56:42.7120838Z dist init r=0, world=2
2023-01-11T20:56:42.7121048Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:56:42.7121389Z   test_diff_hyperparams_sharding_strategy_str_shard_grad_op (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2023-01-11T20:56:42.7121855Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15152
2023-01-11T20:56:42.7122256Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15153
2023-01-11T20:56:42.7122712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7123030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7123456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7123805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7124128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:42.7124595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7124923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7125345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7125673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7126003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:42.7126494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7127015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7127390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:42.7127741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:42.7128002Z dist init r=1, world=2
2023-01-11T20:56:42.7128170Z dist init r=0, world=2
2023-01-11T20:56:42.7128378Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:56:42.7128692Z   test_diff_trainability (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2023-01-11T20:56:42.7129129Z Tests FSDP parity with DDP when using multiple parameter groups and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15180
2023-01-11T20:56:42.7129511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15181
2023-01-11T20:56:42.7129965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7130331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7130742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7131244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7131580Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:42.7132051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7132365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7132790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7133132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7133512Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:42.7134008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7134524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7134911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:42.7135243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:42.7135501Z dist init r=0, world=2
2023-01-11T20:56:42.7135685Z dist init r=1, world=2
2023-01-11T20:56:42.7135879Z skip: Need at least 2 CUDA devices (1.612s)
2023-01-11T20:56:42.7136193Z   test_multiple_optimizers (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2023-01-11T20:56:42.7136619Z Tests using two optimizers where only one sets gradients to ``None``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15208
2023-01-11T20:56:42.7137007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15209
2023-01-11T20:56:42.7137504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7137833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7138259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7138599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7138920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:42.7139388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7139711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7140119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7140454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7140784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:42.7141277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7141776Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7142159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:42.7142503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:42.7142758Z dist init r=1, world=2
2023-01-11T20:56:42.7142996Z dist init r=0, world=2
2023-01-11T20:56:42.7143201Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:56:42.7143459Z   test_no_sync (__main__.TestFSDPUseOrigParamsNoSync)
2023-01-11T20:56:42.7143832Z Tests a basic ``no_sync()`` setup by comparing ``use_orig_params=True`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15236
2023-01-11T20:56:42.7144223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15237
2023-01-11T20:56:42.7144673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7144989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7145415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7145760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7146124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:42.7146579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7146902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7147324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7147660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7147978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:42.7148464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7148975Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7149347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:42.7149689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:42.7149942Z dist init r=0, world=2
2023-01-11T20:56:42.7150120Z dist init r=1, world=2
2023-01-11T20:56:42.7150316Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:56:42.7150610Z   test_access_params_after_forward (__main__.TestFSDPUseOrigParamsParamAccess)
2023-01-11T20:56:42.7151029Z Tests that accessing the original parameters after the forward but ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15264
2023-01-11T20:56:42.7151410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15265
2023-01-11T20:56:42.7151855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7152178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7152610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7152939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7153269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:42.7153740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7154049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7154472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7154808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7155134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:42.7155642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7156155Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7156532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:42.7156871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:42.7157116Z dist init r=0, world=2
2023-01-11T20:56:42.7157294Z dist init r=1, world=2
2023-01-11T20:56:42.7157496Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:56:42.7157798Z   test_multiple_forward_offload_params_False (__main__.TestFSDPUseOrigParamsUnshardReshard)
2023-01-11T20:56:42.7158220Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15292
2023-01-11T20:56:42.7158637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15293
2023-01-11T20:56:42.7159086Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7159402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7159822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7160160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7160480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:42.7160941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7161266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7161684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7162011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7162337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:42.7162823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7163340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7163707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:42.7164046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:42.7164301Z dist init r=0, world=2
2023-01-11T20:56:42.7164469Z dist init r=1, world=2
2023-01-11T20:56:42.7164678Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:56:42.7164993Z   test_multiple_forward_offload_params_True (__main__.TestFSDPUseOrigParamsUnshardReshard)
2023-01-11T20:56:42.7165405Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15320
2023-01-11T20:56:42.7165789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15321
2023-01-11T20:56:42.7166236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7166559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7166968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7167307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7167639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:42.7168142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7168453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7168872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7169206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7169518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:42.7170009Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7170524Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7171051Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:42.7171460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:42.7171717Z dist init r=1, world=2
2023-01-11T20:56:42.7171899Z dist init r=0, world=2
2023-01-11T20:56:42.7172095Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:56:42.7172424Z   test_summon_between_two_forwards_offload_params_False (__main__.TestFSDPUseOrigParamsUnshardReshard)
2023-01-11T20:56:42.7172861Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15348
2023-01-11T20:56:42.7173244Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15349
2023-01-11T20:56:42.7173694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7174022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7174456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7174800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7175122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:42.7175588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7175915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7176329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7176672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7177005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:42.7177561Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7178070Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7178456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:42.7178806Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:42.7179070Z dist init r=0, world=2
2023-01-11T20:56:42.7179238Z dist init r=1, world=2
2023-01-11T20:56:42.7179446Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:56:42.7179779Z   test_summon_between_two_forwards_offload_params_True (__main__.TestFSDPUseOrigParamsUnshardReshard)
2023-01-11T20:56:42.7180197Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15376
2023-01-11T20:56:42.7180586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15377
2023-01-11T20:56:42.7181091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7181404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7181834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7182179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7182523Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:42.7182981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7183316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7183741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7184121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7184445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:42.7184939Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7185463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7185835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:42.7186181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:42.7186437Z dist init r=0, world=2
2023-01-11T20:56:42.7186620Z dist init r=1, world=2
2023-01-11T20:56:42.7186816Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T20:56:42.7187093Z   test_grad_writeback (__main__.TestFSDPUseOrigParamsWriteback)
2023-01-11T20:56:42.7187602Z Tests that changes to the original parameters' gradients are written ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15404
2023-01-11T20:56:42.7187989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15405
2023-01-11T20:56:42.7188439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7188769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7189194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7189521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7189854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:42.7190322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7190636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7191059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7191395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7191723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:42.7192199Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7192708Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7193095Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:42.7193439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:42.7193718Z dist init r=1, world=2
2023-01-11T20:56:42.7193898Z dist init r=0, world=2
2023-01-11T20:56:42.7194101Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:56:42.7194364Z   test_param_writeback (__main__.TestFSDPUseOrigParamsWriteback)
2023-01-11T20:56:42.7194755Z Tests that changes to the original parameters are written back. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15432
2023-01-11T20:56:42.7195144Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15433
2023-01-11T20:56:42.7195600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7195912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7196336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7196720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7197038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:42.7197497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7197819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7198240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7198561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7198888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:42.7199372Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7199887Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7200260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:42.7200604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:42.7200859Z dist init r=0, world=2
2023-01-11T20:56:42.7201024Z dist init r=1, world=2
2023-01-11T20:56:42.7201227Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T20:56:42.7201605Z   test_writeback_shape_mismatch (__main__.TestFSDPUseOrigParamsWriteback) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15460
2023-01-11T20:56:42.7202007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15461
2023-01-11T20:56:42.7202454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7202780Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7203204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7203532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7203860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:56:42.7204323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:56:42.7204643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:56:42.7205049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:56:42.7205390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:56:42.7205720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:56:42.7206232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7206749Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T20:56:42.7207129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:56:42.7207473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:56:42.7207718Z dist init r=0, world=2
2023-01-11T20:56:42.7207898Z dist init r=1, world=2
2023-01-11T20:56:42.7208103Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T20:56:42.7208238Z 
2023-01-11T20:56:42.7208426Z ----------------------------------------------------------------------
2023-01-11T20:56:42.7208664Z Ran 19 tests in 31.225s
2023-01-11T20:56:42.7208778Z 
2023-01-11T20:56:42.7208849Z OK (skipped=19)
2023-01-11T20:56:42.7208955Z 
2023-01-11T20:56:42.7209076Z Generating XML reports...
2023-01-11T20:56:42.7209556Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsFQNs-20230111205611.xml
2023-01-11T20:56:42.7210263Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsMultipleParamGroups-20230111205611.xml
2023-01-11T20:56:42.7211104Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsNoSync-20230111205611.xml
2023-01-11T20:56:42.7211786Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsParamAccess-20230111205611.xml
2023-01-11T20:56:42.7212470Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsUnshardReshard-20230111205611.xml
2023-01-11T20:56:42.7213146Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsWriteback-20230111205611.xml
2023-01-11T20:56:42.7213456Z 
2023-01-11T20:56:42.7213784Z ##[endgroup]
2023-01-11T20:56:42.7214247Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_use_orig_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_use_orig_params_aujsokfw)
2023-01-11T20:56:42.7214518Z 
2023-01-11T20:56:42.7214719Z Running distributed/_tensor/test_tensor_ops ... [2023-01-11 20:56:42.702748]
2023-01-11T20:56:42.7215228Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:56:42.703106]
2023-01-11T20:57:20.0699098Z 
2023-01-11T20:57:20.0716737Z Expand the folded group to see the log file of distributed/_tensor/test_tensor_ops
2023-01-11T20:57:20.0717815Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tensor_ops_yx7oabun)
2023-01-11T20:57:20.0720135Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_tensor_ops
2023-01-11T20:57:20.0720480Z 
2023-01-11T20:57:20.0720731Z Running tests...
2023-01-11T20:57:20.0721246Z ----------------------------------------------------------------------
2023-01-11T20:57:20.0721873Z   test_aten_contiguous (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15499
2023-01-11T20:57:20.0722504Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15500
2023-01-11T20:57:20.0723054Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15501
2023-01-11T20:57:20.0723564Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15502
2023-01-11T20:57:20.0724368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0724702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0725333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0725670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0725993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:57:20.0726456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0726786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0727201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0727542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0727862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:57:20.0728371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0728699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0729126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0729469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0729775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:57:20.0730231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0730555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0731202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0731557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0731882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:57:20.0732243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:57:20.0732594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:57:20.0732959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:57:20.0733322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:57:20.0733819Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0734323Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0734839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0735348Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0735613Z ok (2.172s)
2023-01-11T20:57:20.0735910Z   test_clone (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15555
2023-01-11T20:57:20.0736278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15556
2023-01-11T20:57:20.0736601Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15557
2023-01-11T20:57:20.0736909Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15558
2023-01-11T20:57:20.0737407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0737740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0738250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0738579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0738900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:57:20.0739352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0739661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0740082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0740426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0740750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:57:20.0741243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0741573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0741998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0742326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0742644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:57:20.0743097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0743423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0743833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0744175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0744498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:57:20.0744856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:57:20.0745205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:57:20.0745696Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0746090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:57:20.0746436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:57:20.0746919Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0747433Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0747944Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0748211Z ok (1.720s)
2023-01-11T20:57:20.0748515Z   test_contiguous (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15611
2023-01-11T20:57:20.0748889Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15612
2023-01-11T20:57:20.0749212Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15613
2023-01-11T20:57:20.0749519Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15614
2023-01-11T20:57:20.0749959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0750285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0750733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0751078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0751394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:57:20.0751850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0752159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0752582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0752919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0753223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:57:20.0753711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0754038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0754460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0754784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0755100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:57:20.0755550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0755868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0756278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0756613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0756935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:57:20.0757279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:57:20.0757640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:57:20.0757996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:57:20.0758480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0758862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:57:20.0759342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0759853Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0760363Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0760629Z ok (1.821s)
2023-01-11T20:57:20.0760930Z   test_detach (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15667
2023-01-11T20:57:20.0761296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15668
2023-01-11T20:57:20.0761608Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15669
2023-01-11T20:57:20.0761927Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15670
2023-01-11T20:57:20.0762372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0762697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0763146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0763486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0763804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:57:20.0764252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0764572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0764995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0765336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0765639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:57:20.0766126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0766453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0766871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0767197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0767517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:57:20.0767970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0768277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0768694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0769035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0769359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:57:20.0769705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:57:20.0770069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:57:20.0770426Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:57:20.0771047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0771445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:57:20.0771942Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0772454Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0772954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0773236Z ok (1.720s)
2023-01-11T20:57:20.0773543Z   test_empty_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15723
2023-01-11T20:57:20.0774126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15724
2023-01-11T20:57:20.0774439Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15725
2023-01-11T20:57:20.0774758Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15726
2023-01-11T20:57:20.0775205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0775537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0775950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0776377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0776698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:57:20.0777141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0777552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0777982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0778325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0778630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:57:20.0779136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0779467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0779876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0780213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0780534Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:57:20.0780990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0781295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0781714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0782053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0782362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:57:20.0782722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:57:20.0783083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:57:20.0783440Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:57:20.0783782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:57:20.0784268Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0784786Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0785300Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0785794Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0786075Z ok (1.719s)
2023-01-11T20:57:20.0786379Z   test_fill_inplace (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15779
2023-01-11T20:57:20.0786757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15780
2023-01-11T20:57:20.0787069Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15781
2023-01-11T20:57:20.0787388Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15782
2023-01-11T20:57:20.0787829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0788142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0788567Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0788945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0789268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:57:20.0789713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0790036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0790458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0790784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0791099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:57:20.0791603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0791934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0792341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0792678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0792996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:57:20.0793453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0793764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0794186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0794527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0794834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:57:20.0795191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:57:20.0795555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:57:20.0795916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:57:20.0796391Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0796785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:57:20.0797271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0797783Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0798284Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0798566Z ok (1.720s)
2023-01-11T20:57:20.0798879Z   test_fill_inplace_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15835
2023-01-11T20:57:20.0799248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15836
2023-01-11T20:57:20.0799571Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15837
2023-01-11T20:57:20.0799888Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15838
2023-01-11T20:57:20.0800331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0800644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0801064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0801439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0801742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:57:20.0802195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0802513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0802934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0803259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0803575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:57:20.0804056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0804380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0804789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0805125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0805439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:57:20.0805878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0806197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0806616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0806956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0807266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:57:20.0807624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:57:20.0807987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:57:20.0808334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:57:20.0808692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:57:20.0809176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0809686Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0810187Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0810691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0811104Z ok (1.719s)
2023-01-11T20:57:20.0811411Z   test_full_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15891
2023-01-11T20:57:20.0811769Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15892
2023-01-11T20:57:20.0812099Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15893
2023-01-11T20:57:20.0812423Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15894
2023-01-11T20:57:20.0812859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0813189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0813618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0814029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0814336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:57:20.0814797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0815121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0815549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0815878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0816197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:57:20.0816693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0817013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0817516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0817857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0818175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:57:20.0818614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0818937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0819358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0819685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0820003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:57:20.0820364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:57:20.0820731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:57:20.0821076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:57:20.0821434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:57:20.0821916Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0822429Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0822930Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0823439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0823721Z ok (1.719s)
2023-01-11T20:57:20.0824006Z   test_index (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15947
2023-01-11T20:57:20.0824374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15948
2023-01-11T20:57:20.0824701Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 15949
2023-01-11T20:57:20.0825026Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 15950
2023-01-11T20:57:20.0825461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0825787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0826210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0826593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0826901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:57:20.0827358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0827688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0828101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0828442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0828760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:57:20.0829216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0829559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0829983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0830325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0830631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:57:20.0831084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0831409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0831834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0832163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0832482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:57:20.0832842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:57:20.0833210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:57:20.0833556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:57:20.0834043Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0834441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:57:20.0834916Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0835422Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0835943Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0836230Z ok (9.136s)
2023-01-11T20:57:20.0836522Z   test_inplace_op (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16039
2023-01-11T20:57:20.0836894Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16040
2023-01-11T20:57:20.0837219Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16041
2023-01-11T20:57:20.0837543Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16042
2023-01-11T20:57:20.0837976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0838301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0838727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0839091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0839408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:57:20.0839862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0840181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0840587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0840926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0841243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:57:20.0841681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0842068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0842490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0842833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0843141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:57:20.0843596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0843915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0844338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0844663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0844985Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:57:20.0845349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:57:20.0845703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:57:20.0846062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:57:20.0846418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:57:20.0846904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0847408Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0847917Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0848432Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0849020Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:57:20.0849426Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:57:20.0849970Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:57:20.0850377Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:57:20.0851037Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:57:20.0851504Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:57:20.0852051Z /opt/conda/lib/python3.7/site-packages/torch/distributed/_tensor/device_mesh.py:423: UserWarning: ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!
2023-01-11T20:57:20.0852464Z   "ProcessGroupGloo does not support reduce_scatter, falling back with all reduce!"
2023-01-11T20:57:20.0852706Z ok (1.721s)
2023-01-11T20:57:20.0852998Z   test_ones_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16095
2023-01-11T20:57:20.0853374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16096
2023-01-11T20:57:20.0853699Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16097
2023-01-11T20:57:20.0854008Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16098
2023-01-11T20:57:20.0854488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0854817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0855240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0855565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0855880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:57:20.0856331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0856639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0857054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0857460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0857788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:57:20.0858238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0858563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0858982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0859374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0859679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:57:20.0860134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0860458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0860870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0861208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0861526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:57:20.0861885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:57:20.0862235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:57:20.0862590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:57:20.0862942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:57:20.0863414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0863924Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0864477Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0864982Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0865250Z ok (1.719s)
2023-01-11T20:57:20.0865561Z   test_ones_like_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16151
2023-01-11T20:57:20.0865938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16152
2023-01-11T20:57:20.0866261Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16153
2023-01-11T20:57:20.0866567Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16154
2023-01-11T20:57:20.0867040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0867369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0867777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0868121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0868439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:57:20.0868889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0869199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0869623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0869960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0870281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:57:20.0870722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0871044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0871467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0871791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0872109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:57:20.0872557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0872878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0873286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0873625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0873943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:57:20.0874290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:57:20.0874650Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:57:20.0875006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:57:20.0875358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:57:20.0875830Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0876341Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0876885Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0877390Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0877657Z ok (1.719s)
2023-01-11T20:57:20.0877962Z   test_op_out_variant (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16207
2023-01-11T20:57:20.0878337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16208
2023-01-11T20:57:20.0878645Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16209
2023-01-11T20:57:20.0878962Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16210
2023-01-11T20:57:20.0879443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0879769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0880172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0880512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0880830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:57:20.0881280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0881591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0882015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0882356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0882669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:57:20.0883122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0883448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0883872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0884197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0884516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:57:20.0884968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0885279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0885708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0886050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0886370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:57:20.0886718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:57:20.0887082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:57:20.0887444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:57:20.0887801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:57:20.0888277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0888792Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0889344Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0889851Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0890123Z ok (1.820s)
2023-01-11T20:57:20.0890432Z   test_zero_inplace (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16263
2023-01-11T20:57:20.0890809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16264
2023-01-11T20:57:20.0891256Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16265
2023-01-11T20:57:20.0891581Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16266
2023-01-11T20:57:20.0892104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0892437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0892849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0893190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0893515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:57:20.0893958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0894281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0894704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0895045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0895352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:57:20.0895805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0896123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0896530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0896867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0897181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:57:20.0897708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0898017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0898440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0898781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0899101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:57:20.0899445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:57:20.0899807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:57:20.0900173Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:57:20.0900521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:57:20.0901011Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0901524Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0902086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0902579Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0902865Z ok (1.820s)
2023-01-11T20:57:20.0903166Z   test_zeros_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16319
2023-01-11T20:57:20.0903537Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16320
2023-01-11T20:57:20.0903845Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16321
2023-01-11T20:57:20.0904161Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16322
2023-01-11T20:57:20.0904642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0904962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0905383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0905720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0906037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:57:20.0906476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0906794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0907217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0907538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0907857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:57:20.0908308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0908629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0909039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0909377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0909695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:57:20.0910143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0910450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0910870Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0911211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0911513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:57:20.0911870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:57:20.0912234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:57:20.0912590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:57:20.0912939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:57:20.0913423Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0913939Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0914489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0914982Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0915263Z ok (1.820s)
2023-01-11T20:57:20.0915575Z   test_zeros_like_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16375
2023-01-11T20:57:20.0915947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16376
2023-01-11T20:57:20.0916272Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16377
2023-01-11T20:57:20.0916591Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16378
2023-01-11T20:57:20.0917081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0917394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0917813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0918152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0918456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:57:20.0918909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0919233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0919652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0919978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0920300Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:57:20.0920755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0921086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0921492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0921836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0922153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:57:20.0922595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:57:20.0922915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:57:20.0923336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:57:20.0923674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:57:20.0923976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:57:20.0924330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:57:20.0924694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:57:20.0925038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:57:20.0925392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:57:20.0925875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0926390Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0926927Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0927437Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:57:20.0927719Z ok (1.720s)
2023-01-11T20:57:20.0927821Z 
2023-01-11T20:57:20.0928020Z ----------------------------------------------------------------------
2023-01-11T20:57:20.0928246Z Ran 16 tests in 35.787s
2023-01-11T20:57:20.0928358Z 
2023-01-11T20:57:20.0928419Z OK
2023-01-11T20:57:20.0928511Z 
2023-01-11T20:57:20.0928595Z Generating XML reports...
2023-01-11T20:57:20.0929027Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_tensor_ops/TEST-DistTensorOpsTest-20230111205643.xml
2023-01-11T20:57:20.0929280Z 
2023-01-11T20:57:20.0929745Z ##[endgroup]
2023-01-11T20:57:20.0930230Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tensor_ops_yx7oabun)
2023-01-11T20:57:20.0930488Z 
2023-01-11T20:57:20.0930674Z Running distributed/fsdp/test_wrap ... [2023-01-11 20:57:20.070964]
2023-01-11T20:57:20.0931292Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:57:20.071316]
2023-01-11T20:58:02.7485154Z 
2023-01-11T20:58:02.7486271Z Expand the folded group to see the log file of distributed/fsdp/test_wrap
2023-01-11T20:58:02.7487418Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_wrap_qrmehya8)
2023-01-11T20:58:02.7488380Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_wrap
2023-01-11T20:58:02.7488730Z 
2023-01-11T20:58:02.7488863Z Running tests...
2023-01-11T20:58:02.7528241Z ----------------------------------------------------------------------
2023-01-11T20:58:02.7528747Z   test_always_wrap (__main__.TestAutoWrap)
2023-01-11T20:58:02.7529184Z Test to ensure that if `always_wrap_policy` is ... skip: Test Requires CUDA (0.001s)
2023-01-11T20:58:02.7529905Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2023-01-11T20:58:02.7530680Z   test_always_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2023-01-11T20:58:02.7531407Z   test_auto_wrap_api (__main__.TestAutoWrap)
2023-01-11T20:58:02.7531849Z Test to ensure with auto wrap, we wrap child modules correctly based on the min_num_params. ... skip: Requires at least 2 GPUs (0.000s)
2023-01-11T20:58:02.7532355Z   test_auto_wrap_preset_exclude_wrap (__main__.TestAutoWrap)
2023-01-11T20:58:02.7532956Z Test to ensure excluded modules are not wrapped, regardless if the total param size is greater than the ... skip: Requires at least 2 GPUs (0.001s)
2023-01-11T20:58:02.7533555Z   test_auto_wrap_preset_exclude_wrap_include_children (__main__.TestAutoWrap)
2023-01-11T20:58:02.7534228Z Test to ensure excluded modules are not wrapped, but children are if param size is greater than ... skip: Requires at least 2 GPUs (0.000s)
2023-01-11T20:58:02.7534691Z   test_auto_wrap_preset_force_leaf (__main__.TestAutoWrap)
2023-01-11T20:58:02.7535194Z Test to ensure force-leaf modules are not wrapped, and children are not wrapped. The ... skip: Requires at least 2 GPUs (0.001s)
2023-01-11T20:58:02.7535558Z   test_auto_wrap_preset_force_leaf_custom (__main__.TestAutoWrap)
2023-01-11T20:58:02.7535947Z Test to ensure force-leaf modules are not wrapped. ... skip: Requires at least 2 GPUs (0.001s)
2023-01-11T20:58:02.7536396Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s)
2023-01-11T20:58:02.7536907Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s)
2023-01-11T20:58:02.7537673Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s)
2023-01-11T20:58:02.7538190Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_AFTER_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s)
2023-01-11T20:58:02.7538710Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s)
2023-01-11T20:58:02.7539294Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=False)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s)
2023-01-11T20:58:02.7539798Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_False (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s)
2023-01-11T20:58:02.7540308Z   test_auto_wrap_smoke_test_cuda_init_mode_CUDAInitMode_CUDA_BEFORE_cpu_offload_CPUOffload(offload_params=True)_use_device_id_True (__main__.TestAutoWrap) ... skip: Test Requires CUDA (0.001s)
2023-01-11T20:58:02.7540772Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2023-01-11T20:58:02.7541194Z   test_auto_wrap_with_ignored_modules_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2023-01-11T20:58:02.7541507Z   test_module_wrap_policy (__main__.TestAutoWrap)
2023-01-11T20:58:02.7541785Z Tests the ``ModuleWrapPolicy``. ... skip: Requires at least 2 GPUs (0.000s)
2023-01-11T20:58:02.7542081Z   test_transformer_auto_wrap_policy (__main__.TestAutoWrap)
2023-01-11T20:58:02.7542381Z Tests the ``transformer_auto_wrap_policy``. ... skip: Requires at least 2 GPUs (0.000s)
2023-01-11T20:58:02.7542712Z   test_wrap_disabled_outside_context (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2023-01-11T20:58:02.7543059Z   test_wrap_override_defaults (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.000s)
2023-01-11T20:58:02.7543411Z   test_wrap_wrap_method_WrapMethod_FSDP_CTOR (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2023-01-11T20:58:02.7543767Z   test_wrap_wrap_method_WrapMethod_WRAP_API (__main__.TestAutoWrap) ... skip: Requires at least 2 GPUs (0.001s)
2023-01-11T20:58:02.7544087Z   test_bn_always_wrapped_individually (__main__.TestFSDPWrap)
2023-01-11T20:58:02.7544472Z Ensures that by using _or_policy with _wrap_batchnorm_individually, even ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16442
2023-01-11T20:58:02.7544870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16443
2023-01-11T20:58:02.7545181Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16444
2023-01-11T20:58:02.7545498Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16445
2023-01-11T20:58:02.7545965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7546293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7546705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7547047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7547381Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7547839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7551653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7554991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7555353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7555926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7556719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7557042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7557789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7558291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7559114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7560217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7560821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7561709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7565818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7566389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7571595Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7572605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7573514Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7582055Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7582481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7582836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7583165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7583500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7583763Z dist init r=0, world=4
2023-01-11T20:58:02.7583929Z dist init r=3, world=4
2023-01-11T20:58:02.7584109Z dist init r=1, world=4
2023-01-11T20:58:02.7584284Z dist init r=2, world=4
2023-01-11T20:58:02.7584485Z skip: Need at least 2 CUDA devices (1.970s)
2023-01-11T20:58:02.7584798Z   test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2023-01-11T20:58:02.7585225Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16498
2023-01-11T20:58:02.7585620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16499
2023-01-11T20:58:02.7585934Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16500
2023-01-11T20:58:02.7586253Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16501
2023-01-11T20:58:02.7586722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7587038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7587466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7587966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7588305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7588763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7589091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7589519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7589860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7590177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7590691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7591019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7591429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7591767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7592099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7592560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7592868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7593293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7593635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7593959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7594453Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7594969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7595787Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7596282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7596661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7597004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7597346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7597671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7597921Z dist init r=3, world=4
2023-01-11T20:58:02.7598101Z dist init r=1, world=4
2023-01-11T20:58:02.7598265Z dist init r=2, world=4
2023-01-11T20:58:02.7598440Z dist init r=0, world=4
2023-01-11T20:58:02.7598646Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T20:58:02.7598944Z   test_error_already_wrapped_nested_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2023-01-11T20:58:02.7599368Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16554
2023-01-11T20:58:02.7599760Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16555
2023-01-11T20:58:02.7600087Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16556
2023-01-11T20:58:02.7600436Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16557
2023-01-11T20:58:02.7600888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7601213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7601636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7601965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7602299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7602762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7603070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7603524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7603869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7604199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7604650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7604972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7605396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7605738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7606051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7606542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7607037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7607347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7607767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7608107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7608437Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7608914Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7609427Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7609943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7610322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7610649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7611233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7611573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7611819Z dist init r=0, world=4
2023-01-11T20:58:02.7612001Z dist init r=3, world=4
2023-01-11T20:58:02.7612181Z dist init r=2, world=4
2023-01-11T20:58:02.7612344Z dist init r=1, world=4
2023-01-11T20:58:02.7612551Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T20:58:02.7612864Z   test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap)
2023-01-11T20:58:02.7613293Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16610
2023-01-11T20:58:02.7613755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16611
2023-01-11T20:58:02.7614084Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16612
2023-01-11T20:58:02.7614408Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16613
2023-01-11T20:58:02.7614857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7615189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7615614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7615956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7616316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7616786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7617107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7617584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7617912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7618247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7618716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7619027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7619453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7619799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7620131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7620583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7620907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7621328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7621653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7621983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7622473Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7622998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7623499Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7624006Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7624387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7624729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7625057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7625391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7625646Z dist init r=3, world=4
2023-01-11T20:58:02.7625852Z dist init r=1, world=4
2023-01-11T20:58:02.7626027Z dist init r=0, world=4
2023-01-11T20:58:02.7626202Z dist init r=2, world=4
2023-01-11T20:58:02.7626394Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T20:58:02.7626703Z   test_error_already_wrapped_nested_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap)
2023-01-11T20:58:02.7627128Z Test that an error is raised if we attempt to wrap when submodules are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16666
2023-01-11T20:58:02.7627524Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16667
2023-01-11T20:58:02.7627834Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16668
2023-01-11T20:58:02.7628147Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16669
2023-01-11T20:58:02.7628604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7628968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7629380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7629723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7630054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7630504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7630822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7631245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7631584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7631903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7632368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7632696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7633107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7633450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7633785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7634274Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7634775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7635269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7635595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7636017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7636343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7636673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7637164Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7637678Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7638046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7638395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7638765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7639091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7639348Z dist init r=0, world=4
2023-01-11T20:58:02.7639531Z dist init r=1, world=4
2023-01-11T20:58:02.7639693Z dist init r=3, world=4
2023-01-11T20:58:02.7639873Z dist init r=2, world=4
2023-01-11T20:58:02.7640077Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T20:58:02.7640586Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16722
2023-01-11T20:58:02.7641188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16723
2023-01-11T20:58:02.7641539Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16724
2023-01-11T20:58:02.7641863Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16725
2023-01-11T20:58:02.7642319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7642633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7643065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7643409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7643748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7644195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7644521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7644943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7645280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7645594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7646056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7646376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7646782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7647119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7647446Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7647909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7648216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7648634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7648969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7649281Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7649769Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7650283Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7650799Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7651475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7651859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7652206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7652551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7652879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7653131Z dist init r=0, world=4
2023-01-11T20:58:02.7653315Z dist init r=2, world=4
2023-01-11T20:58:02.7653479Z dist init r=3, world=4
2023-01-11T20:58:02.7653657Z dist init r=1, world=4
2023-01-11T20:58:02.7653862Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T20:58:02.7654438Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16778
2023-01-11T20:58:02.7654981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16779
2023-01-11T20:58:02.7655311Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16780
2023-01-11T20:58:02.7655639Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16781
2023-01-11T20:58:02.7656076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7656409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7656834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7657180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7657591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7658064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7658393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7658803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7659147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7659485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7659981Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7660466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7660796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7661217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7661560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7661877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7662342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7662670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7663080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7663416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7663797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7664285Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7664788Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7665301Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7665684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7666035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7666363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7666732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7666994Z dist init r=3, world=4
2023-01-11T20:58:02.7667162Z dist init r=0, world=4
2023-01-11T20:58:02.7667339Z dist init r=2, world=4
2023-01-11T20:58:02.7667516Z dist init r=1, world=4
2023-01-11T20:58:02.7667705Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T20:58:02.7668214Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16834
2023-01-11T20:58:02.7668746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16835
2023-01-11T20:58:02.7669077Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16836
2023-01-11T20:58:02.7669386Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16837
2023-01-11T20:58:02.7669837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7670164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7670589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7670917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7671249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7671710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7672021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7672442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7672783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7673115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7673564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7673886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7674305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7674642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7674956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7675420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7675740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7676183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7676520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7676851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7677339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7677840Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7678355Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7678864Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7679275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7679612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7679945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7680281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7680523Z dist init r=1, world=4
2023-01-11T20:58:02.7680700Z dist init r=0, world=4
2023-01-11T20:58:02.7680880Z dist init r=2, world=4
2023-01-11T20:58:02.7681042Z dist init r=3, world=4
2023-01-11T20:58:02.7681247Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T20:58:02.7681750Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16890
2023-01-11T20:58:02.7682284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16891
2023-01-11T20:58:02.7682600Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16892
2023-01-11T20:58:02.7682918Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16893
2023-01-11T20:58:02.7683367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7683693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7684103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7684442Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7684776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7685227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7685555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7685978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7686321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7686640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7687106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7687429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7687852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7688177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7688545Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7689034Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7689513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7689836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7690255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7690595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7691057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7691612Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7692138Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7692649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7693016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7693363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7693704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7694035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7694292Z dist init r=3, world=4
2023-01-11T20:58:02.7694476Z dist init r=1, world=4
2023-01-11T20:58:02.7694655Z dist init r=2, world=4
2023-01-11T20:58:02.7694819Z dist init r=0, world=4
2023-01-11T20:58:02.7695031Z skip: Need at least 2 CUDA devices (1.721s)
2023-01-11T20:58:02.7695537Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16946
2023-01-11T20:58:02.7696054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16947
2023-01-11T20:58:02.7696377Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 16948
2023-01-11T20:58:02.7696700Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 16949
2023-01-11T20:58:02.7697150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7697526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7697960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7698305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7698626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7699088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7699414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7699836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7700161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7700497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7700964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7701334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7701741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7702077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7702406Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7702854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7703173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7703596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7703936Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7704323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7704825Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7705341Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7705853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7706345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7706727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7707073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7707403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7707743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7707992Z dist init r=0, world=4
2023-01-11T20:58:02.7708272Z dist init r=2, world=4
2023-01-11T20:58:02.7708439Z dist init r=3, world=4
2023-01-11T20:58:02.7708616Z dist init r=1, world=4
2023-01-11T20:58:02.7708824Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T20:58:02.7709319Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17002
2023-01-11T20:58:02.7709852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17003
2023-01-11T20:58:02.7710185Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17004
2023-01-11T20:58:02.7710517Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17005
2023-01-11T20:58:02.7710962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7711288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7711711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7712041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7712376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7712837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7713159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7713599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7713988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7714319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7714782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7715093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7715523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7715866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7716182Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7716647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7717011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7717437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7717759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7718090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7718583Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7719100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7719597Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7720114Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7720497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7720827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7721165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7721497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7721756Z dist init r=3, world=4
2023-01-11T20:58:02.7721921Z dist init r=1, world=4
2023-01-11T20:58:02.7722094Z dist init r=2, world=4
2023-01-11T20:58:02.7722268Z dist init r=0, world=4
2023-01-11T20:58:02.7722459Z skip: Need at least 2 CUDA devices (1.822s)
2023-01-11T20:58:02.7722962Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17058
2023-01-11T20:58:02.7723490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17059
2023-01-11T20:58:02.7723816Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17060
2023-01-11T20:58:02.7724132Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17061
2023-01-11T20:58:02.7724575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7724899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7725310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7725653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7725987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7726486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7726797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7727219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7727561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7727892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7728364Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7728854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7729179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7729622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7729965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7730295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7730756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7731220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7731651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7731993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7732310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7732805Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7733326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7733841Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7734207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7734553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7734889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7735232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7735473Z dist init r=0, world=4
2023-01-11T20:58:02.7735653Z dist init r=1, world=4
2023-01-11T20:58:02.7735835Z dist init r=3, world=4
2023-01-11T20:58:02.7735998Z dist init r=2, world=4
2023-01-11T20:58:02.7736202Z skip: Need at least 2 CUDA devices (1.721s)
2023-01-11T20:58:02.7736708Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=False)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17114
2023-01-11T20:58:02.7737233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17115
2023-01-11T20:58:02.7737615Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17116
2023-01-11T20:58:02.7737938Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17117
2023-01-11T20:58:02.7738392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7738713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7739204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7739549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7739882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7740335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7740661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7741084Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7741427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7741745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7742274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7742601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7743009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7743351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7743687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7744181Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7744662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7744988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7745420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7745748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7746080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7746568Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7747086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7747583Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7747961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7748308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7748652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7748978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7749230Z dist init r=2, world=4
2023-01-11T20:58:02.7749411Z dist init r=0, world=4
2023-01-11T20:58:02.7749573Z dist init r=1, world=4
2023-01-11T20:58:02.7749750Z dist init r=3, world=4
2023-01-11T20:58:02.7749956Z skip: Need at least 2 CUDA devices (1.721s)
2023-01-11T20:58:02.7750448Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17170
2023-01-11T20:58:02.7750978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17171
2023-01-11T20:58:02.7751310Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17172
2023-01-11T20:58:02.7751679Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17173
2023-01-11T20:58:02.7752118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7752442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7752867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7753207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7753527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7753987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7754310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7754751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7755093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7755421Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7755883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7756194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7756617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7756957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7757288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7757744Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7758065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7758492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7758815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7759145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7759643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7760159Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7760664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7761180Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7761561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7761906Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7762234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7762567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7762820Z dist init r=0, world=4
2023-01-11T20:58:02.7762985Z dist init r=1, world=4
2023-01-11T20:58:02.7763161Z dist init r=3, world=4
2023-01-11T20:58:02.7763336Z dist init r=2, world=4
2023-01-11T20:58:02.7763527Z skip: Need at least 2 CUDA devices (1.722s)
2023-01-11T20:58:02.7764035Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17226
2023-01-11T20:58:02.7764606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17227
2023-01-11T20:58:02.7764932Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17228
2023-01-11T20:58:02.7765240Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17229
2023-01-11T20:58:02.7765685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7766007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7766429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7766793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7767127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7767599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7767908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7768330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7768670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7769005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7769483Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7769976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7770302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7770725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7771163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7771500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7771969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7772279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7772710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7773058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7773393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7773875Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7774392Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7774905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7775284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7775613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7775949Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7776294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7776606Z dist init r=2, world=4
2023-01-11T20:58:02.7776790Z dist init r=1, world=4
2023-01-11T20:58:02.7776968Z dist init r=3, world=4
2023-01-11T20:58:02.7777132Z dist init r=0, world=4
2023-01-11T20:58:02.7777343Z skip: Need at least 2 CUDA devices (1.722s)
2023-01-11T20:58:02.7777925Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17282
2023-01-11T20:58:02.7778460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17283
2023-01-11T20:58:02.7778773Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17284
2023-01-11T20:58:02.7779096Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17285
2023-01-11T20:58:02.7779625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7779957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7780370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7780716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7781052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7781506Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7781833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7782260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7782605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7782927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7783397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7783720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7784147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7784470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7784801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7785261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7785569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7785992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7786333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7786663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7787137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7787654Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7788166Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7788672Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7789045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7789428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7789772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7790099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7790358Z dist init r=2, world=4
2023-01-11T20:58:02.7790537Z dist init r=3, world=4
2023-01-11T20:58:02.7790713Z dist init r=1, world=4
2023-01-11T20:58:02.7790875Z dist init r=0, world=4
2023-01-11T20:58:02.7825513Z skip: Need at least 2 CUDA devices (1.722s)
2023-01-11T20:58:02.7826111Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_POST_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17338
2023-01-11T20:58:02.7826789Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17339
2023-01-11T20:58:02.7827105Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17340
2023-01-11T20:58:02.7827419Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17341
2023-01-11T20:58:02.7827926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7828241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7828652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7828982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7829310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7829771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7830087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7830500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7830837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7831162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7831643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7832122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7832430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7832841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7833167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7833481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7833930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7834242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7834650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7834994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7835322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7835801Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7836372Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7836876Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7837244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7837571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7837898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7838233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7838482Z dist init r=0, world=4
2023-01-11T20:58:02.7838644Z dist init r=2, world=4
2023-01-11T20:58:02.7838809Z dist init r=1, world=4
2023-01-11T20:58:02.7838971Z dist init r=3, world=4
2023-01-11T20:58:02.7839193Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T20:58:02.7839693Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17394
2023-01-11T20:58:02.7840225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17395
2023-01-11T20:58:02.7840541Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17396
2023-01-11T20:58:02.7840870Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17397
2023-01-11T20:58:02.7841317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7841642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7842055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7842403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7842736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7843202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7843515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7843941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7844283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7844601Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7845066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7845397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7845821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7846148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7846481Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7846976Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7847467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7847779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7848203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7848586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7848906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7849396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7849911Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7850426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7850795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7851266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7851674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7852025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7852274Z dist init r=3, world=4
2023-01-11T20:58:02.7852457Z dist init r=1, world=4
2023-01-11T20:58:02.7852640Z dist init r=2, world=4
2023-01-11T20:58:02.7852804Z dist init r=0, world=4
2023-01-11T20:58:02.7853011Z skip: Need at least 2 CUDA devices (1.922s)
2023-01-11T20:58:02.7853526Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_False_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17450
2023-01-11T20:58:02.7854045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17451
2023-01-11T20:58:02.7854374Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17452
2023-01-11T20:58:02.7854700Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17453
2023-01-11T20:58:02.7855154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7855469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7855895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7856236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7856572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7857026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7857347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7857841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7858172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7858505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7858976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7859303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7859716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7860062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7860397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7860845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7861226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7861650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7861990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7862305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7862802Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7863323Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7863839Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7864367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7864756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7865102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7865442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7865772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7866027Z dist init r=0, world=4
2023-01-11T20:58:02.7866208Z dist init r=3, world=4
2023-01-11T20:58:02.7866372Z dist init r=2, world=4
2023-01-11T20:58:02.7866545Z dist init r=1, world=4
2023-01-11T20:58:02.7866749Z skip: Need at least 2 CUDA devices (1.722s)
2023-01-11T20:58:02.7867243Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_AFTER (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17506
2023-01-11T20:58:02.7867768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17507
2023-01-11T20:58:02.7868096Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17508
2023-01-11T20:58:02.7868424Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17509
2023-01-11T20:58:02.7868862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7869190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7869616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7869960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7870277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7870750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7871075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7871488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7871826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7872158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7872626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7872939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7873365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7873752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7874070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7874534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7874859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7875284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7875608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7875937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7876429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7876983Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7877482Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7877990Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7878372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7878714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7879045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7879386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7879642Z dist init r=3, world=4
2023-01-11T20:58:02.7879808Z dist init r=2, world=4
2023-01-11T20:58:02.7879989Z dist init r=0, world=4
2023-01-11T20:58:02.7880164Z dist init r=1, world=4
2023-01-11T20:58:02.7880354Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T20:58:02.7880859Z   test_main_wrap_api_cpu_offload_CPUOffload(offload_params=True)_backward_prefetch_BackwardPrefetch_BACKWARD_PRE_forward_prefetch_True_cuda_init_mode_CUDAInitMode_CUDA_BEFORE (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17562
2023-01-11T20:58:02.7881388Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17563
2023-01-11T20:58:02.7881716Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17564
2023-01-11T20:58:02.7882028Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17565
2023-01-11T20:58:02.7882471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7882803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7883216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7883559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7883888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7884354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7884664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7885087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7885428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7885760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7886253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7886580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7887006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7887331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7887656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7888118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7888441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7888848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7889222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7889556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7890049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7890551Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7891171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7891691Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7892060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7892410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7892756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7893104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7893347Z dist init r=1, world=4
2023-01-11T20:58:02.7893533Z dist init r=2, world=4
2023-01-11T20:58:02.7893714Z dist init r=3, world=4
2023-01-11T20:58:02.7893878Z dist init r=0, world=4
2023-01-11T20:58:02.7894084Z skip: Need at least 2 CUDA devices (1.822s)
2023-01-11T20:58:02.7894459Z   test_wrap_batchnorm_individually_use_or_policy_False (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17618
2023-01-11T20:58:02.7894866Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17619
2023-01-11T20:58:02.7895182Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17620
2023-01-11T20:58:02.7895506Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17621
2023-01-11T20:58:02.7895961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7896277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7896702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7897046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7897379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7897904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7898231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7898660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7899051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7899388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7899858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7900187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7900599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7900945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7901282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7901774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7902297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7902626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7903056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7903383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7903715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7904204Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7904721Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7905217Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7905599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7905947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7906295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7906738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7906997Z dist init r=1, world=4
2023-01-11T20:58:02.7907182Z dist init r=3, world=4
2023-01-11T20:58:02.7907347Z dist init r=2, world=4
2023-01-11T20:58:02.7907530Z dist init r=0, world=4
2023-01-11T20:58:02.7907738Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T20:58:02.7908098Z   test_wrap_batchnorm_individually_use_or_policy_True (__main__.TestFSDPWrap) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17674
2023-01-11T20:58:02.7908508Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17675
2023-01-11T20:58:02.7908848Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 17676
2023-01-11T20:58:02.7909179Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 17677
2023-01-11T20:58:02.7909623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7909955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7910385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7910732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7911057Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T20:58:02.7911535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7911913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7912327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7912677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7913017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T20:58:02.7913514Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7913998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7914325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7914743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7915125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7915443Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T20:58:02.7915941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T20:58:02.7916278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T20:58:02.7916706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T20:58:02.7917040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T20:58:02.7917373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T20:58:02.7917872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7918399Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7918908Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T20:58:02.7919298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T20:58:02.7919643Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T20:58:02.7919984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T20:58:02.7920315Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T20:58:02.7920570Z dist init r=3, world=4
2023-01-11T20:58:02.7920751Z dist init r=2, world=4
2023-01-11T20:58:02.7920916Z dist init r=0, world=4
2023-01-11T20:58:02.7921095Z dist init r=1, world=4
2023-01-11T20:58:02.7921302Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T20:58:02.7921441Z 
2023-01-11T20:58:02.7921633Z ----------------------------------------------------------------------
2023-01-11T20:58:02.7921875Z Ran 47 tests in 41.056s
2023-01-11T20:58:02.7921988Z 
2023-01-11T20:58:02.7922060Z OK (skipped=47)
2023-01-11T20:58:02.7922167Z 
2023-01-11T20:58:02.7922252Z Generating XML reports...
2023-01-11T20:58:02.7922663Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20230111205721.xml
2023-01-11T20:58:02.7923196Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20230111205721.xml
2023-01-11T20:58:02.7923436Z 
2023-01-11T20:58:02.7923785Z ##[endgroup]
2023-01-11T20:58:02.7924201Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_wrap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_wrap_qrmehya8)
2023-01-11T20:58:02.7924451Z 
2023-01-11T20:58:02.7924635Z Running distributed/test_c10d_nccl ... [2023-01-11 20:58:02.749970]
2023-01-11T20:58:02.7925212Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 20:58:02.750317]
2023-01-11T21:02:01.5277784Z 
2023-01-11T21:02:01.5278259Z Expand the folded group to see the log file of distributed/test_c10d_nccl
2023-01-11T21:02:01.5279213Z ##[group]PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_nccl_o6fdxhvm)
2023-01-11T21:02:01.5282246Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_all_reduce_coalesced_nccl>, <__main__.CommTest testMethod=test_broadcast_coalesced_nccl>, <__main__.CommTest testMethod=test_nccl_barrier>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids>, <__main__.CommTest testMethod=test_nccl_barrier_device_ids_function_argument>, <__main__.CommTest testMethod=test_nccl_barrier_timeout>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group>, <__main__.CommTest testMethod=test_nccl_barrier_timeout_new_group_non_member>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_detail>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_info>, <__main__.CommTest testMethod=test_nccl_warn_not_in_group_debug_off>, <__main__.CommTest testMethod=test_nncl_rank_membership>, <__main__.CommTest testMethod=test_pass_nccl_options_high_priority_stream>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_nccl_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_nccl>, <__main__.CommTest testMethod=test_sequence_num_set_nccl_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]>
2023-01-11T21:02:01.5287486Z test_all_reduce_coalesced_nccl (__main__.CommTest)
2023-01-11T21:02:01.5287896Z test_broadcast_coalesced_nccl (__main__.CommTest)
2023-01-11T21:02:01.5288421Z test_nccl_barrier (__main__.CommTest)
2023-01-11T21:02:01.5289120Z test_nccl_barrier_device_ids (__main__.CommTest)
2023-01-11T21:02:01.5289658Z test_nccl_barrier_device_ids_function_argument (__main__.CommTest)
2023-01-11T21:02:01.5290041Z test_nccl_barrier_timeout (__main__.CommTest)
2023-01-11T21:02:01.5290536Z test_nccl_barrier_timeout_new_group (__main__.CommTest)
2023-01-11T21:02:01.5291219Z test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest)
2023-01-11T21:02:01.5291626Z test_nccl_warn_not_in_group_debug_detail (__main__.CommTest)
2023-01-11T21:02:01.5292149Z test_nccl_warn_not_in_group_debug_info (__main__.CommTest)
2023-01-11T21:02:01.5292653Z test_nccl_warn_not_in_group_debug_off (__main__.CommTest)
2023-01-11T21:02:01.5293010Z test_nncl_rank_membership (__main__.CommTest)
2023-01-11T21:02:01.5293523Z test_pass_nccl_options_high_priority_stream (__main__.CommTest)
2023-01-11T21:02:01.5293936Z test_sequence_num_incremented_nccl_default (__main__.CommTest)
2023-01-11T21:02:01.5294473Z test_sequence_num_incremented_nccl_subgroup (__main__.CommTest)
2023-01-11T21:02:01.5294984Z test_sequence_num_set_default_pg_nccl (__main__.CommTest)
2023-01-11T21:02:01.5295390Z test_sequence_num_set_nccl_new_group (__main__.CommTest)
2023-01-11T21:02:01.5295896Z test_tensor_dtype_complex (__main__.CommTest)
2023-01-11T21:02:01.5296239Z test_tensor_dtype_mismatch (__main__.CommTest)
2023-01-11T21:02:01.5297768Z <unittest.suite.TestSuite tests=[<__main__.CompilerTest testMethod=test_allgather_into_tensor_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allgather_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_alltoall_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_reduce_scatter_tensor_work_wait_gpu>, <__main__.CompilerTest testMethod=test_reduce_scatter_work_wait_gpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]>
2023-01-11T21:02:01.5299845Z test_allgather_into_tensor_work_wait_gpu (__main__.CompilerTest)
2023-01-11T21:02:01.5300274Z test_allgather_work_wait_gpu (__main__.CompilerTest)
2023-01-11T21:02:01.5300662Z test_allreduce_work_wait_gpu (__main__.CompilerTest)
2023-01-11T21:02:01.5301029Z test_alltoall_work_wait_gpu (__main__.CompilerTest)
2023-01-11T21:02:01.5301419Z test_broadcast_work_wait_gpu (__main__.CompilerTest)
2023-01-11T21:02:01.5301819Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest)
2023-01-11T21:02:01.5302229Z test_nested_comm_tensor_wrapping (__main__.CompilerTest)
2023-01-11T21:02:01.5302650Z test_reduce_scatter_tensor_work_wait_gpu (__main__.CompilerTest)
2023-01-11T21:02:01.5303078Z test_reduce_scatter_work_wait_gpu (__main__.CompilerTest)
2023-01-11T21:02:01.5303471Z test_scatter_work_wait_gpu (__main__.CompilerTest)
2023-01-11T21:02:01.5315130Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module>, <__main__.DistributedDataParallelTest testMethod=test_accumulate_gradients_module_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value>, <__main__.DistributedDataParallelTest testMethod=test_arbitrary_forward_return_value_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_bf16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_builtin_ddp_comm_hooks_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_channels_last_contig>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_hook_nccl_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_allreduce_with_then_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_nccl>, <__main__.DistributedDataParallelTest testMethod=test_ddp_multi_device_module_config>, <__main__.DistributedDataParallelTest testMethod=test_ddp_packed_sequence>, <__main__.DistributedDataParallelTest testMethod=test_ddp_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_with_lazy_parameters>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl>, <__main__.DistributedDataParallelTest testMethod=test_default_ddp_comm_hooks_nccl_is_view>, <__main__.DistributedDataParallelTest testMethod=test_failure_recovery>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_detail>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_info>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_kwarg_grad_is_view_debug_off>, <__main__.DistributedDataParallelTest testMethod=test_fp16>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_is_view>, <__main__.DistributedDataParallelTest testMethod=test_fp16_compress_wrapper_nccl>, <__main__.DistributedDataParallelTest testMethod=test_fp16_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_1devicemodule_1replicaperprocess>, <__main__.DistributedDataParallelTest testMethod=test_grad_layout_2devicemodule>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward>, <__main__.DistributedDataParallelTest testMethod=test_multiple_outputs_multiple_backward_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_ids_not_allowed>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_multi_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_device_ids_None>, <__main__.DistributedDataParallelTest testMethod=test_nccl_backend_single_device_module_empty_device_ids>, <__main__.DistributedDataParallelTest testMethod=test_nccl_propagate_error_reason>, <__main__.DistributedDataParallelTest testMethod=test_no_grad>, <__main__.DistributedDataParallelTest testMethod=test_param_layout_mismatch_error>, <__main__.DistributedDataParallelTest testMethod=test_pass_default_pg>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl>, <__main__.DistributedDataParallelTest testMethod=test_powerSGD_ddp_comm_hook_nccl_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]>
2023-01-11T21:02:01.5326535Z test_accumulate_gradients_module (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5327119Z test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5327640Z test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5328190Z test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5328690Z test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5329130Z test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5329567Z test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5330015Z test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5330471Z test_channels_last_contig (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5331060Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5331565Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5332214Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5332708Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5333254Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5333813Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5334389Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5334946Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5335470Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5335990Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5336664Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5337228Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5337788Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5338382Z test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5338883Z test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5339401Z test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5339896Z test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5340391Z test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5340882Z test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5341353Z test_ddp_packed_sequence (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5341806Z test_ddp_weight_sharing (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5342275Z test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5342745Z test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5343262Z test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5343732Z test_failure_recovery (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5344217Z test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5344703Z test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5345219Z test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5345747Z test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5346271Z test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5346765Z test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5347188Z test_fp16 (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5347612Z test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5348067Z test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5348506Z test_fp16_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5348980Z test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5349461Z test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5349913Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5350426Z test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5351114Z test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5351605Z test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5351977Z test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5352323Z test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5352633Z test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5352952Z test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5353303Z test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5353669Z test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5354067Z test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5354415Z test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5354714Z test_no_grad (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5355010Z test_param_layout_mismatch_error (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5355303Z test_pass_default_pg (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5355607Z test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5355940Z test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5356255Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5356573Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5356850Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:02:01.5357821Z <unittest.suite.TestSuite tests=[<__main__.NcclErrorHandlingTest testMethod=test_invalid_nccl_blocking_wait_env>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_blocking_wait_with_barrier>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_abort>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_clean_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_nonzero_exit>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigkill>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_blocking_sigterm>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_errors_nonblocking>, <__main__.NcclErrorHandlingTest testMethod=test_nccl_timeout>]>
2023-01-11T21:02:01.5358771Z test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest)
2023-01-11T21:02:01.5359061Z test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest)
2023-01-11T21:02:01.5359355Z test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest)
2023-01-11T21:02:01.5359651Z test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest)
2023-01-11T21:02:01.5359954Z test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest)
2023-01-11T21:02:01.5360238Z test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest)
2023-01-11T21:02:01.5360531Z test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest)
2023-01-11T21:02:01.5360820Z test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest)
2023-01-11T21:02:01.5361075Z test_nccl_timeout (__main__.NcclErrorHandlingTest)
2023-01-11T21:02:01.5361993Z <unittest.suite.TestSuite tests=[<__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_all_to_all_single>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_allgather_base>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_allreduce_coalesced>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_collectives>, <__main__.NcclProcessGroupWithDispatchedCollectivesTests testMethod=test_reduce_scatter_base>]>
2023-01-11T21:02:01.5362980Z test_all_to_all_single (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2023-01-11T21:02:01.5363375Z test_allgather_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2023-01-11T21:02:01.5363759Z test_allreduce_coalesced (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2023-01-11T21:02:01.5364154Z test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2023-01-11T21:02:01.5364545Z test_reduce_scatter_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests)
2023-01-11T21:02:01.5364947Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLNoGPUTest testMethod=test_init_no_gpus>]>
2023-01-11T21:02:01.5365275Z test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest)
2023-01-11T21:02:01.5366851Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allgather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_allreduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_barrier>, <__main__.ProcessGroupNCCLTest testMethod=test_broadcast_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_empty_tensors>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_gather_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_nccl_dist_backend_error>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_basics>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_base_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_reduce_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_checks>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_ops>, <__main__.ProcessGroupNCCLTest testMethod=test_scatter_stress>, <__main__.ProcessGroupNCCLTest testMethod=test_send_recv>]>
2023-01-11T21:02:01.5368468Z test_allgather_base_basics (__main__.ProcessGroupNCCLTest)
2023-01-11T21:02:01.5368749Z test_allgather_base_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T21:02:01.5369016Z test_allgather_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T21:02:01.5369276Z test_allreduce_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T21:02:01.5369518Z test_barrier (__main__.ProcessGroupNCCLTest)
2023-01-11T21:02:01.5369770Z test_broadcast_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T21:02:01.5370027Z test_empty_tensors (__main__.ProcessGroupNCCLTest)
2023-01-11T21:02:01.5370272Z test_gather_checks (__main__.ProcessGroupNCCLTest)
2023-01-11T21:02:01.5370528Z test_gather_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T21:02:01.5370783Z test_gather_stress (__main__.ProcessGroupNCCLTest)
2023-01-11T21:02:01.5371196Z test_nccl_dist_backend_error (__main__.ProcessGroupNCCLTest)
2023-01-11T21:02:01.5371465Z test_reduce_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T21:02:01.5371743Z test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest)
2023-01-11T21:02:01.5372022Z test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T21:02:01.5372299Z test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T21:02:01.5372566Z test_scatter_checks (__main__.ProcessGroupNCCLTest)
2023-01-11T21:02:01.5372829Z test_scatter_ops (__main__.ProcessGroupNCCLTest)
2023-01-11T21:02:01.5373074Z test_scatter_stress (__main__.ProcessGroupNCCLTest)
2023-01-11T21:02:01.5373329Z test_send_recv (__main__.ProcessGroupNCCLTest)
2023-01-11T21:02:01.5373641Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_common_errors>]>
2023-01-11T21:02:01.5373933Z test_common_errors (__main__.RendezvousEnvTest)
2023-01-11T21:02:01.5374221Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:02:01.5374756Z <unittest.suite.TestSuite tests=[<__main__.TimeoutTest testMethod=test_default_store_timeout_nccl>]>
2023-01-11T21:02:01.5375108Z test_default_store_timeout_nccl (__main__.TimeoutTest)
2023-01-11T21:02:01.5375689Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5376162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5376499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5376916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5377264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5377429Z 
2023-01-11T21:02:01.5377503Z Running tests...
2023-01-11T21:02:01.5377796Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5378208Z   test_all_reduce_coalesced_nccl (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5378410Z 
2023-01-11T21:02:01.5378653Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5378895Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5378996Z 
2023-01-11T21:02:01.5379069Z OK (skipped=1)
2023-01-11T21:02:01.5379179Z 
2023-01-11T21:02:01.5379263Z Generating XML reports...
2023-01-11T21:02:01.5379669Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205805.xml
2023-01-11T21:02:01.5380107Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5380566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5380900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5381325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5381656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5381819Z 
2023-01-11T21:02:01.5381897Z Running tests...
2023-01-11T21:02:01.5382196Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5382529Z   test_broadcast_coalesced_nccl (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5382726Z 
2023-01-11T21:02:01.5382917Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5383152Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5383262Z 
2023-01-11T21:02:01.5383333Z OK (skipped=1)
2023-01-11T21:02:01.5383440Z 
2023-01-11T21:02:01.5383511Z Generating XML reports...
2023-01-11T21:02:01.5383914Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205806.xml
2023-01-11T21:02:01.5384360Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5384819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5385138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5385566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5385908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5386070Z 
2023-01-11T21:02:01.5386130Z Running tests...
2023-01-11T21:02:01.5386427Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5386754Z   test_nccl_barrier (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5386939Z 
2023-01-11T21:02:01.5387132Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5387354Z Ran 1 test in 0.002s
2023-01-11T21:02:01.5387462Z 
2023-01-11T21:02:01.5387534Z OK (skipped=1)
2023-01-11T21:02:01.5387639Z 
2023-01-11T21:02:01.5387724Z Generating XML reports...
2023-01-11T21:02:01.5388116Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205808.xml
2023-01-11T21:02:01.5388596Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5389048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5389372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5389786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5390127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5390288Z 
2023-01-11T21:02:01.5390360Z Running tests...
2023-01-11T21:02:01.5390646Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5390989Z   test_nccl_barrier_device_ids (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5391191Z 
2023-01-11T21:02:01.5391434Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5391671Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5391769Z 
2023-01-11T21:02:01.5391840Z OK (skipped=1)
2023-01-11T21:02:01.5391943Z 
2023-01-11T21:02:01.5392027Z Generating XML reports...
2023-01-11T21:02:01.5392425Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205809.xml
2023-01-11T21:02:01.5392867Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5393321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5393645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5394071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5394410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5394572Z 
2023-01-11T21:02:01.5394644Z Running tests...
2023-01-11T21:02:01.5394949Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5395306Z   test_nccl_barrier_device_ids_function_argument (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5395506Z 
2023-01-11T21:02:01.5395699Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5395941Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5396051Z 
2023-01-11T21:02:01.5396123Z OK (skipped=1)
2023-01-11T21:02:01.5396231Z 
2023-01-11T21:02:01.5396304Z Generating XML reports...
2023-01-11T21:02:01.5396708Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205811.xml
2023-01-11T21:02:01.5397160Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5397621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5397937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5398364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5398709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5398870Z 
2023-01-11T21:02:01.5398931Z Running tests...
2023-01-11T21:02:01.5399235Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5399574Z   test_nccl_barrier_timeout (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5399768Z 
2023-01-11T21:02:01.5399962Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5400188Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5400300Z 
2023-01-11T21:02:01.5400372Z OK (skipped=1)
2023-01-11T21:02:01.5400511Z 
2023-01-11T21:02:01.5400596Z Generating XML reports...
2023-01-11T21:02:01.5400987Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205812.xml
2023-01-11T21:02:01.5401442Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5401902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5402234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5402647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5402994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5403157Z 
2023-01-11T21:02:01.5403231Z Running tests...
2023-01-11T21:02:01.5403521Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5403897Z   test_nccl_barrier_timeout_new_group (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5404105Z 
2023-01-11T21:02:01.5404298Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5404533Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5404644Z 
2023-01-11T21:02:01.5404703Z OK (skipped=1)
2023-01-11T21:02:01.5404809Z 
2023-01-11T21:02:01.5404891Z Generating XML reports...
2023-01-11T21:02:01.5405291Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205813.xml
2023-01-11T21:02:01.5405737Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5406183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5406511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5406936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5407269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5407433Z 
2023-01-11T21:02:01.5407504Z Running tests...
2023-01-11T21:02:01.5407801Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5408164Z   test_nccl_barrier_timeout_new_group_non_member (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5408366Z 
2023-01-11T21:02:01.5408556Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5408790Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5408899Z 
2023-01-11T21:02:01.5408969Z OK (skipped=1)
2023-01-11T21:02:01.5409074Z 
2023-01-11T21:02:01.5409145Z Generating XML reports...
2023-01-11T21:02:01.5409547Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205815.xml
2023-01-11T21:02:01.5409993Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5410451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5410761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5411361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5411707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5411871Z 
2023-01-11T21:02:01.5411930Z Running tests...
2023-01-11T21:02:01.5412229Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5412577Z   test_nccl_warn_not_in_group_debug_detail (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5412790Z 
2023-01-11T21:02:01.5412985Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5413274Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5413384Z 
2023-01-11T21:02:01.5413456Z OK (skipped=1)
2023-01-11T21:02:01.5413565Z 
2023-01-11T21:02:01.5413649Z Generating XML reports...
2023-01-11T21:02:01.5414039Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205816.xml
2023-01-11T21:02:01.5414483Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5414936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5415264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5415678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5416020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5416185Z 
2023-01-11T21:02:01.5416259Z Running tests...
2023-01-11T21:02:01.5416597Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5416929Z   test_nccl_warn_not_in_group_debug_info (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5417132Z 
2023-01-11T21:02:01.5417323Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5417563Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5417672Z 
2023-01-11T21:02:01.5417731Z OK (skipped=1)
2023-01-11T21:02:01.5417834Z 
2023-01-11T21:02:01.5417920Z Generating XML reports...
2023-01-11T21:02:01.5418391Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205818.xml
2023-01-11T21:02:01.5418842Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5419289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5419623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5420045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5420375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5420536Z 
2023-01-11T21:02:01.5420610Z Running tests...
2023-01-11T21:02:01.5420909Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5421258Z   test_nccl_warn_not_in_group_debug_off (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5421461Z 
2023-01-11T21:02:01.5421640Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5421874Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5421987Z 
2023-01-11T21:02:01.5422057Z OK (skipped=1)
2023-01-11T21:02:01.5422163Z 
2023-01-11T21:02:01.5422234Z Generating XML reports...
2023-01-11T21:02:01.5422635Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205819.xml
2023-01-11T21:02:01.5424132Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5424589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5424901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5425327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5425670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5425834Z 
2023-01-11T21:02:01.5425906Z Running tests...
2023-01-11T21:02:01.5426191Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5426524Z   test_nncl_rank_membership (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5426766Z 
2023-01-11T21:02:01.5426960Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5427187Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5427298Z 
2023-01-11T21:02:01.5427368Z OK (skipped=1)
2023-01-11T21:02:01.5427475Z 
2023-01-11T21:02:01.5427559Z Generating XML reports...
2023-01-11T21:02:01.5427960Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205821.xml
2023-01-11T21:02:01.5428394Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5428846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5429174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5429583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5429959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5430125Z 
2023-01-11T21:02:01.5430199Z Running tests...
2023-01-11T21:02:01.5430496Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5430842Z   test_pass_nccl_options_high_priority_stream (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5431053Z 
2023-01-11T21:02:01.5431245Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5431476Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5431589Z 
2023-01-11T21:02:01.5431660Z OK (skipped=1)
2023-01-11T21:02:01.5431753Z 
2023-01-11T21:02:01.5431837Z Generating XML reports...
2023-01-11T21:02:01.5432241Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205822.xml
2023-01-11T21:02:01.5432691Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5433137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5433464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5433888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5434230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5434377Z 
2023-01-11T21:02:01.5434448Z Running tests...
2023-01-11T21:02:01.5434747Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5435106Z   test_sequence_num_incremented_nccl_default (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5435313Z 
2023-01-11T21:02:01.5435496Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5435733Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5435843Z 
2023-01-11T21:02:01.5435917Z OK (skipped=1)
2023-01-11T21:02:01.5436026Z 
2023-01-11T21:02:01.5436109Z Generating XML reports...
2023-01-11T21:02:01.5436497Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205823.xml
2023-01-11T21:02:01.5436944Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5437396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5437710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5438134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5438475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5438635Z 
2023-01-11T21:02:01.5438707Z Running tests...
2023-01-11T21:02:01.5438991Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5439386Z   test_sequence_num_incremented_nccl_subgroup (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5439599Z 
2023-01-11T21:02:01.5439790Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5440026Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5440127Z 
2023-01-11T21:02:01.5440198Z OK (skipped=1)
2023-01-11T21:02:01.5440303Z 
2023-01-11T21:02:01.5440386Z Generating XML reports...
2023-01-11T21:02:01.5440783Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205825.xml
2023-01-11T21:02:01.5441213Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5441666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5441995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5442449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5442781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5442943Z 
2023-01-11T21:02:01.5443017Z Running tests...
2023-01-11T21:02:01.5443314Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5443650Z   test_sequence_num_set_default_pg_nccl (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5443854Z 
2023-01-11T21:02:01.5444047Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5444292Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5444402Z 
2023-01-11T21:02:01.5444477Z OK (skipped=1)
2023-01-11T21:02:01.5444570Z 
2023-01-11T21:02:01.5444658Z Generating XML reports...
2023-01-11T21:02:01.5445062Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205826.xml
2023-01-11T21:02:01.5445514Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5445959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5446294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5446724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5447068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5447217Z 
2023-01-11T21:02:01.5447292Z Running tests...
2023-01-11T21:02:01.5447597Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5447951Z   test_sequence_num_set_nccl_new_group (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5448155Z 
2023-01-11T21:02:01.5448353Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5448579Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5448694Z 
2023-01-11T21:02:01.5448766Z OK (skipped=1)
2023-01-11T21:02:01.5448875Z 
2023-01-11T21:02:01.5448960Z Generating XML reports...
2023-01-11T21:02:01.5449351Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205828.xml
2023-01-11T21:02:01.5449800Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5450261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5450592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5451159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5451504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5451666Z 
2023-01-11T21:02:01.5451803Z Running tests...
2023-01-11T21:02:01.5452097Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5452441Z   test_tensor_dtype_complex (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5452634Z 
2023-01-11T21:02:01.5452832Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5453066Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5453164Z 
2023-01-11T21:02:01.5453233Z OK (skipped=1)
2023-01-11T21:02:01.5453342Z 
2023-01-11T21:02:01.5453424Z Generating XML reports...
2023-01-11T21:02:01.5453829Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205829.xml
2023-01-11T21:02:01.5454268Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5454719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5455091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5455521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5455856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5456017Z 
2023-01-11T21:02:01.5456090Z Running tests...
2023-01-11T21:02:01.5456392Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5456718Z   test_tensor_dtype_mismatch (__main__.CommTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5456913Z 
2023-01-11T21:02:01.5457106Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5457339Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5457452Z 
2023-01-11T21:02:01.5457522Z OK (skipped=1)
2023-01-11T21:02:01.5457616Z 
2023-01-11T21:02:01.5457698Z Generating XML reports...
2023-01-11T21:02:01.5458161Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205831.xml
2023-01-11T21:02:01.5458617Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5459063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5459398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5459822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5460169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5460332Z 
2023-01-11T21:02:01.5460392Z Running tests...
2023-01-11T21:02:01.5460688Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5461080Z   test_allgather_into_tensor_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17941
2023-01-11T21:02:01.5461477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17942
2023-01-11T21:02:01.5461913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5462237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5462662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5463073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5463399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5463857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5464179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5464590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5464976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5465297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5465575Z skip: Need at least 2 CUDA devices (1.761s)
2023-01-11T21:02:01.5465711Z 
2023-01-11T21:02:01.5465908Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5466143Z Ran 1 test in 1.762s
2023-01-11T21:02:01.5466253Z 
2023-01-11T21:02:01.5466326Z OK (skipped=1)
2023-01-11T21:02:01.5466422Z 
2023-01-11T21:02:01.5466508Z Generating XML reports...
2023-01-11T21:02:01.5466925Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205832.xml
2023-01-11T21:02:01.5467384Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5467861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5468191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5468619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5468959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5469120Z 
2023-01-11T21:02:01.5469181Z Running tests...
2023-01-11T21:02:01.5469480Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5469862Z   test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17974
2023-01-11T21:02:01.5470233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17975
2023-01-11T21:02:01.5470679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5471008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5471428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5471765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5472090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5472549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5472870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5473280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5473742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5474070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5474346Z skip: Need at least 2 CUDA devices (1.763s)
2023-01-11T21:02:01.5474481Z 
2023-01-11T21:02:01.5474682Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5474922Z Ran 1 test in 1.763s
2023-01-11T21:02:01.5475033Z 
2023-01-11T21:02:01.5475104Z OK (skipped=1)
2023-01-11T21:02:01.5475198Z 
2023-01-11T21:02:01.5475281Z Generating XML reports...
2023-01-11T21:02:01.5475696Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205835.xml
2023-01-11T21:02:01.5476151Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5476597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5476922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5477385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5477723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5477871Z 
2023-01-11T21:02:01.5477942Z Running tests...
2023-01-11T21:02:01.5478241Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5478622Z   test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18007
2023-01-11T21:02:01.5478993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18008
2023-01-11T21:02:01.5479442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5479768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5480259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5480591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5480910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5481370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5481698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5482107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5482448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5482769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5483048Z skip: Need at least 2 CUDA devices (1.860s)
2023-01-11T21:02:01.5483184Z 
2023-01-11T21:02:01.5483384Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5483625Z Ran 1 test in 1.860s
2023-01-11T21:02:01.5483735Z 
2023-01-11T21:02:01.5483807Z OK (skipped=1)
2023-01-11T21:02:01.5483901Z 
2023-01-11T21:02:01.5483986Z Generating XML reports...
2023-01-11T21:02:01.5484400Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205839.xml
2023-01-11T21:02:01.5484855Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5485292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5485614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5486037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5486378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5486530Z 
2023-01-11T21:02:01.5486603Z Running tests...
2023-01-11T21:02:01.5486907Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5487286Z   test_alltoall_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18040
2023-01-11T21:02:01.5487653Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18041
2023-01-11T21:02:01.5488102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5488424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5488845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5489172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5489492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5489983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5490304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5490717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5491210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5491533Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5491808Z skip: Need at least 2 CUDA devices (1.758s)
2023-01-11T21:02:01.5491943Z 
2023-01-11T21:02:01.5492144Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5492380Z Ran 1 test in 1.758s
2023-01-11T21:02:01.5492491Z 
2023-01-11T21:02:01.5492565Z OK (skipped=1)
2023-01-11T21:02:01.5492659Z 
2023-01-11T21:02:01.5492744Z Generating XML reports...
2023-01-11T21:02:01.5493220Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205842.xml
2023-01-11T21:02:01.5493683Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5494128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5494460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5494886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5495227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5495377Z 
2023-01-11T21:02:01.5495450Z Running tests...
2023-01-11T21:02:01.5495754Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5496143Z   test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18073
2023-01-11T21:02:01.5496517Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18074
2023-01-11T21:02:01.5496969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5497297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5497728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5498122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5498453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5498915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5499245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5499665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5500017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5500343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5500622Z skip: Need at least 2 CUDA devices (1.860s)
2023-01-11T21:02:01.5500759Z 
2023-01-11T21:02:01.5500958Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5501205Z Ran 1 test in 1.861s
2023-01-11T21:02:01.5501317Z 
2023-01-11T21:02:01.5501390Z OK (skipped=1)
2023-01-11T21:02:01.5501485Z 
2023-01-11T21:02:01.5501570Z Generating XML reports...
2023-01-11T21:02:01.5501987Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205845.xml
2023-01-11T21:02:01.5502454Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5502958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5503287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5503712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5504057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5504207Z 
2023-01-11T21:02:01.5504282Z Running tests...
2023-01-11T21:02:01.5504583Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5504975Z   test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18106
2023-01-11T21:02:01.5505352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18107
2023-01-11T21:02:01.5505840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5506174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5506606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5506937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5507256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5507711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5508022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5508444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5508786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5509112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5509388Z skip: Need at least 2 CUDA devices (1.864s)
2023-01-11T21:02:01.5509524Z 
2023-01-11T21:02:01.5509725Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5509961Z Ran 1 test in 1.864s
2023-01-11T21:02:01.5510071Z 
2023-01-11T21:02:01.5510131Z OK (skipped=1)
2023-01-11T21:02:01.5510237Z 
2023-01-11T21:02:01.5510323Z Generating XML reports...
2023-01-11T21:02:01.5510740Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205849.xml
2023-01-11T21:02:01.5511195Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5511637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5511966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5512392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5512733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5512882Z 
2023-01-11T21:02:01.5512955Z Running tests...
2023-01-11T21:02:01.5513255Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5513644Z   test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18139
2023-01-11T21:02:01.5514020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18140
2023-01-11T21:02:01.5514468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5514795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5515222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5515586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5515908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5516362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5516918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5517346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5517690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5518012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5518289Z skip: Need at least 2 CUDA devices (1.865s)
2023-01-11T21:02:01.5518426Z 
2023-01-11T21:02:01.5518658Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5518898Z Ran 1 test in 1.865s
2023-01-11T21:02:01.5519010Z 
2023-01-11T21:02:01.5519070Z OK (skipped=1)
2023-01-11T21:02:01.5519177Z 
2023-01-11T21:02:01.5519262Z Generating XML reports...
2023-01-11T21:02:01.5519681Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205852.xml
2023-01-11T21:02:01.5520140Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5520581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5520908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5521335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5521682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5521834Z 
2023-01-11T21:02:01.5521907Z Running tests...
2023-01-11T21:02:01.5522206Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5522595Z   test_reduce_scatter_tensor_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18172
2023-01-11T21:02:01.5522975Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18173
2023-01-11T21:02:01.5523422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5523746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5524263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5524593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5524915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5525373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5525686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5526112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5526454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5526776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5527051Z skip: Need at least 2 CUDA devices (1.867s)
2023-01-11T21:02:01.5527185Z 
2023-01-11T21:02:01.5527387Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5527624Z Ran 1 test in 1.867s
2023-01-11T21:02:01.5527734Z 
2023-01-11T21:02:01.5527794Z OK (skipped=1)
2023-01-11T21:02:01.5527934Z 
2023-01-11T21:02:01.5528020Z Generating XML reports...
2023-01-11T21:02:01.5528435Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205856.xml
2023-01-11T21:02:01.5528896Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5529334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5529658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5530077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5530415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5530569Z 
2023-01-11T21:02:01.5530640Z Running tests...
2023-01-11T21:02:01.5531149Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5531610Z   test_reduce_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18205
2023-01-11T21:02:01.5531987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18206
2023-01-11T21:02:01.5532442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5532769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5533195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5533525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5533845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5534303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5534618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5535039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5535377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5535698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5535973Z skip: Need at least 2 CUDA devices (1.763s)
2023-01-11T21:02:01.5536108Z 
2023-01-11T21:02:01.5536306Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5536544Z Ran 1 test in 1.763s
2023-01-11T21:02:01.5536655Z 
2023-01-11T21:02:01.5536714Z OK (skipped=1)
2023-01-11T21:02:01.5536818Z 
2023-01-11T21:02:01.5536902Z Generating XML reports...
2023-01-11T21:02:01.5537319Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205859.xml
2023-01-11T21:02:01.5537782Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5538295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5538625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5539051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5539381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5539546Z 
2023-01-11T21:02:01.5539618Z Running tests...
2023-01-11T21:02:01.5539917Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5540295Z   test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18238
2023-01-11T21:02:01.5540662Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18239
2023-01-11T21:02:01.5541162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5541486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5541908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5542236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5542555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5543010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5543319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5543742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5544115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5544437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5544712Z skip: Need at least 2 CUDA devices (1.763s)
2023-01-11T21:02:01.5544846Z 
2023-01-11T21:02:01.5545040Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5545281Z Ran 1 test in 1.763s
2023-01-11T21:02:01.5545392Z 
2023-01-11T21:02:01.5545451Z OK (skipped=1)
2023-01-11T21:02:01.5545556Z 
2023-01-11T21:02:01.5545640Z Generating XML reports...
2023-01-11T21:02:01.5546058Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205903.xml
2023-01-11T21:02:01.5546510Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5546955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5547285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5547706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5548036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5548201Z 
2023-01-11T21:02:01.5548275Z Running tests...
2023-01-11T21:02:01.5548576Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5548965Z   test_accumulate_gradients_module (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5549203Z 
2023-01-11T21:02:01.5549385Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5549626Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5549738Z 
2023-01-11T21:02:01.5549811Z OK (skipped=1)
2023-01-11T21:02:01.5549920Z 
2023-01-11T21:02:01.5550006Z Generating XML reports...
2023-01-11T21:02:01.5550459Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205906.xml
2023-01-11T21:02:01.5550960Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5551420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5551736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5552163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5552507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5552670Z 
2023-01-11T21:02:01.5552744Z Running tests...
2023-01-11T21:02:01.5553030Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5553436Z   test_accumulate_gradients_module_with_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5553719Z 
2023-01-11T21:02:01.5553913Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5554137Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5554252Z 
2023-01-11T21:02:01.5554324Z OK (skipped=1)
2023-01-11T21:02:01.5554430Z 
2023-01-11T21:02:01.5554516Z Generating XML reports...
2023-01-11T21:02:01.5554981Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205907.xml
2023-01-11T21:02:01.5555462Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5555921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5556251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5556699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5557048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5557212Z 
2023-01-11T21:02:01.5557287Z Running tests...
2023-01-11T21:02:01.5557587Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5557956Z   test_arbitrary_forward_return_value (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5558192Z 
2023-01-11T21:02:01.5558384Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5558624Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5558736Z 
2023-01-11T21:02:01.5558807Z OK (skipped=1)
2023-01-11T21:02:01.5558901Z 
2023-01-11T21:02:01.5558984Z Generating XML reports...
2023-01-11T21:02:01.5559448Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205909.xml
2023-01-11T21:02:01.5559948Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5560389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5560715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5561141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5561487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5561636Z 
2023-01-11T21:02:01.5561707Z Running tests...
2023-01-11T21:02:01.5562002Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5562397Z   test_arbitrary_forward_return_value_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5562645Z 
2023-01-11T21:02:01.5562846Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5563073Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5563183Z 
2023-01-11T21:02:01.5563252Z OK (skipped=1)
2023-01-11T21:02:01.5563356Z 
2023-01-11T21:02:01.5563438Z Generating XML reports...
2023-01-11T21:02:01.5563882Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205910.xml
2023-01-11T21:02:01.5564371Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5564824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5565153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5565563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5565905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5566112Z 
2023-01-11T21:02:01.5566184Z Running tests...
2023-01-11T21:02:01.5566473Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5566853Z   test_bf16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5567086Z 
2023-01-11T21:02:01.5567280Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5567512Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5567613Z 
2023-01-11T21:02:01.5567683Z OK (skipped=1)
2023-01-11T21:02:01.5567788Z 
2023-01-11T21:02:01.5567872Z Generating XML reports...
2023-01-11T21:02:01.5568329Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205912.xml
2023-01-11T21:02:01.5568804Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5569302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5569633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5570057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5570389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5570551Z 
2023-01-11T21:02:01.5570625Z Running tests...
2023-01-11T21:02:01.5571085Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5571459Z   test_bf16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5571694Z 
2023-01-11T21:02:01.5571896Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5572131Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5572242Z 
2023-01-11T21:02:01.5572314Z OK (skipped=1)
2023-01-11T21:02:01.5572425Z 
2023-01-11T21:02:01.5572499Z Generating XML reports...
2023-01-11T21:02:01.5572967Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205913.xml
2023-01-11T21:02:01.5573463Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5573920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5574236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5574660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5575006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5575167Z 
2023-01-11T21:02:01.5575226Z Running tests...
2023-01-11T21:02:01.5575522Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5575904Z   test_builtin_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5576136Z 
2023-01-11T21:02:01.5576330Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5576558Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5576667Z 
2023-01-11T21:02:01.5576739Z OK (skipped=1)
2023-01-11T21:02:01.5576845Z 
2023-01-11T21:02:01.5576932Z Generating XML reports...
2023-01-11T21:02:01.5577379Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205914.xml
2023-01-11T21:02:01.5577880Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5578391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5578720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5579207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5579554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5579721Z 
2023-01-11T21:02:01.5579795Z Running tests...
2023-01-11T21:02:01.5580098Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5580474Z   test_builtin_ddp_comm_hooks_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5580719Z 
2023-01-11T21:02:01.5580916Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5581156Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5581268Z 
2023-01-11T21:02:01.5581328Z OK (skipped=1)
2023-01-11T21:02:01.5581434Z 
2023-01-11T21:02:01.5581522Z Generating XML reports...
2023-01-11T21:02:01.5582025Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205916.xml
2023-01-11T21:02:01.5582525Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5582975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5583305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5583736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5584070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5584235Z 
2023-01-11T21:02:01.5584306Z Running tests...
2023-01-11T21:02:01.5584603Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5584973Z   test_channels_last_contig (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5585202Z 
2023-01-11T21:02:01.5585384Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5585622Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5585735Z 
2023-01-11T21:02:01.5585809Z OK (skipped=1)
2023-01-11T21:02:01.5585912Z 
2023-01-11T21:02:01.5605325Z Generating XML reports...
2023-01-11T21:02:01.5605948Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205917.xml
2023-01-11T21:02:01.5606461Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5606925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5607262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5607680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5608038Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5608205Z 
2023-01-11T21:02:01.5608277Z Running tests...
2023-01-11T21:02:01.5608583Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5608899Z   test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5609436Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18361
2023-01-11T21:02:01.5609844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18362
2023-01-11T21:02:01.5610282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5610612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5611221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5611702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5612013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5612471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5612797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5613212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5613555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5613877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5614166Z skip: Need at least 2 CUDA devices (1.762s)
2023-01-11T21:02:01.5614304Z 
2023-01-11T21:02:01.5614492Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5614783Z Ran 1 test in 1.762s
2023-01-11T21:02:01.5614897Z 
2023-01-11T21:02:01.5614972Z OK (skipped=1)
2023-01-11T21:02:01.5615078Z 
2023-01-11T21:02:01.5615163Z Generating XML reports...
2023-01-11T21:02:01.5615618Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205919.xml
2023-01-11T21:02:01.5616122Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5616581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5616895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5617317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5617659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5617821Z 
2023-01-11T21:02:01.5617898Z Running tests...
2023-01-11T21:02:01.5618245Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5618589Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5619016Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18394
2023-01-11T21:02:01.5619406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18395
2023-01-11T21:02:01.5619858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5620184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5620610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5620940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5621267Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5621724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5622048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5622460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5622800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5623121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5623396Z skip: Need at least 2 CUDA devices (1.759s)
2023-01-11T21:02:01.5623531Z 
2023-01-11T21:02:01.5623728Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5623967Z Ran 1 test in 1.760s
2023-01-11T21:02:01.5624079Z 
2023-01-11T21:02:01.5624191Z OK (skipped=1)
2023-01-11T21:02:01.5624289Z 
2023-01-11T21:02:01.5624373Z Generating XML reports...
2023-01-11T21:02:01.5624836Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205922.xml
2023-01-11T21:02:01.5625334Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5625778Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5626104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5626530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5626875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5627026Z 
2023-01-11T21:02:01.5627099Z Running tests...
2023-01-11T21:02:01.5627404Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5627779Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5628187Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18427
2023-01-11T21:02:01.5628581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18428
2023-01-11T21:02:01.5629034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5629364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5629780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5630124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5630450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5630897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5631227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5631652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5631996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5632302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5632591Z skip: Need at least 2 CUDA devices (1.762s)
2023-01-11T21:02:01.5632725Z 
2023-01-11T21:02:01.5632924Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5633164Z Ran 1 test in 1.762s
2023-01-11T21:02:01.5633263Z 
2023-01-11T21:02:01.5633338Z OK (skipped=1)
2023-01-11T21:02:01.5633443Z 
2023-01-11T21:02:01.5633530Z Generating XML reports...
2023-01-11T21:02:01.5634000Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205925.xml
2023-01-11T21:02:01.5634488Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5634951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5635283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5635711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5636044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5636206Z 
2023-01-11T21:02:01.5636278Z Running tests...
2023-01-11T21:02:01.5636578Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5636902Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5637355Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18460
2023-01-11T21:02:01.5637751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18461
2023-01-11T21:02:01.5638200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5638513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5638937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5639280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5639588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5640072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5640403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5640833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5641161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5641481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5641771Z skip: Need at least 2 CUDA devices (1.759s)
2023-01-11T21:02:01.5641904Z 
2023-01-11T21:02:01.5642099Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5642325Z Ran 1 test in 1.759s
2023-01-11T21:02:01.5642437Z 
2023-01-11T21:02:01.5642510Z OK (skipped=1)
2023-01-11T21:02:01.5642616Z 
2023-01-11T21:02:01.5642700Z Generating XML reports...
2023-01-11T21:02:01.5643151Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205929.xml
2023-01-11T21:02:01.5643650Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5644106Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5644434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5644844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5645192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5645354Z 
2023-01-11T21:02:01.5645429Z Running tests...
2023-01-11T21:02:01.5645717Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5646071Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5646606Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18493
2023-01-11T21:02:01.5647006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18494
2023-01-11T21:02:01.5647446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5647774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5648204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5648545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5648854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5649308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5649634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5650080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5650423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5650740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5651219Z skip: Need at least 2 CUDA devices (1.760s)
2023-01-11T21:02:01.5651343Z 
2023-01-11T21:02:01.5651551Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5651790Z Ran 1 test in 1.760s
2023-01-11T21:02:01.5651902Z 
2023-01-11T21:02:01.5651974Z OK (skipped=1)
2023-01-11T21:02:01.5652082Z 
2023-01-11T21:02:01.5652153Z Generating XML reports...
2023-01-11T21:02:01.5652624Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205932.xml
2023-01-11T21:02:01.5653187Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5653651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5653965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5654388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5654732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5654893Z 
2023-01-11T21:02:01.5654953Z Running tests...
2023-01-11T21:02:01.5655251Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5655607Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5656142Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18526
2023-01-11T21:02:01.5656530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18527
2023-01-11T21:02:01.5656975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5657301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5657713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5658059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5658445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5658902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5659215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5659644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5659991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5660311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5660584Z skip: Need at least 2 CUDA devices (1.762s)
2023-01-11T21:02:01.5660719Z 
2023-01-11T21:02:01.5660915Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5661156Z Ran 1 test in 1.762s
2023-01-11T21:02:01.5661267Z 
2023-01-11T21:02:01.5661326Z OK (skipped=1)
2023-01-11T21:02:01.5661432Z 
2023-01-11T21:02:01.5661516Z Generating XML reports...
2023-01-11T21:02:01.5661979Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205935.xml
2023-01-11T21:02:01.5662478Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5662972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5663298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5663722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5664053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5664215Z 
2023-01-11T21:02:01.5664289Z Running tests...
2023-01-11T21:02:01.5664589Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5664925Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5665451Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18559
2023-01-11T21:02:01.5665903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18560
2023-01-11T21:02:01.5666361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5666686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5667099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5667438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5667758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5668199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5668519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5668941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5669281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5669592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5669880Z skip: Need at least 2 CUDA devices (1.761s)
2023-01-11T21:02:01.5670013Z 
2023-01-11T21:02:01.5670210Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5670433Z Ran 1 test in 1.761s
2023-01-11T21:02:01.5670546Z 
2023-01-11T21:02:01.5670617Z OK (skipped=1)
2023-01-11T21:02:01.5670725Z 
2023-01-11T21:02:01.5670811Z Generating XML reports...
2023-01-11T21:02:01.5671277Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205939.xml
2023-01-11T21:02:01.5671763Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5672222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5672548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5672962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5673302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5673465Z 
2023-01-11T21:02:01.5673537Z Running tests...
2023-01-11T21:02:01.5673835Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5674162Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5674698Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18592
2023-01-11T21:02:01.5675102Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18593
2023-01-11T21:02:01.5675552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5675898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5676323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5676662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5676971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5677428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5677752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5678173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5678499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5678852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5679142Z skip: Need at least 2 CUDA devices (1.765s)
2023-01-11T21:02:01.5679268Z 
2023-01-11T21:02:01.5679472Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5679709Z Ran 1 test in 1.765s
2023-01-11T21:02:01.5679819Z 
2023-01-11T21:02:01.5679890Z OK (skipped=1)
2023-01-11T21:02:01.5679984Z 
2023-01-11T21:02:01.5680068Z Generating XML reports...
2023-01-11T21:02:01.5680532Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205942.xml
2023-01-11T21:02:01.5681031Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5681472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5681797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5682222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5682566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5682728Z 
2023-01-11T21:02:01.5682788Z Running tests...
2023-01-11T21:02:01.5683087Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5683419Z   test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5683836Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18625
2023-01-11T21:02:01.5684239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18626
2023-01-11T21:02:01.5684684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5685015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5685427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5685766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5686086Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5686538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5686851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5687274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5687613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5687918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5688237Z skip: Need at least 2 CUDA devices (1.756s)
2023-01-11T21:02:01.5688370Z 
2023-01-11T21:02:01.5688575Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5688814Z Ran 1 test in 1.756s
2023-01-11T21:02:01.5688912Z 
2023-01-11T21:02:01.5688983Z OK (skipped=1)
2023-01-11T21:02:01.5689088Z 
2023-01-11T21:02:01.5689172Z Generating XML reports...
2023-01-11T21:02:01.5689634Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205945.xml
2023-01-11T21:02:01.5690129Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5690588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5691047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5691529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5691861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5692026Z 
2023-01-11T21:02:01.5692100Z Running tests...
2023-01-11T21:02:01.5692402Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5692740Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5693184Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18658
2023-01-11T21:02:01.5693589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18659
2023-01-11T21:02:01.5694039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5694358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5694792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5695134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5695456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5695899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5696222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5696642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5696971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5697287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5697579Z skip: Need at least 2 CUDA devices (1.860s)
2023-01-11T21:02:01.5697715Z 
2023-01-11T21:02:01.5697912Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5698199Z Ran 1 test in 1.860s
2023-01-11T21:02:01.5698309Z 
2023-01-11T21:02:01.5698381Z OK (skipped=1)
2023-01-11T21:02:01.5698489Z 
2023-01-11T21:02:01.5698573Z Generating XML reports...
2023-01-11T21:02:01.5699031Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205949.xml
2023-01-11T21:02:01.5699532Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5699986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5700316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5700727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5701118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5701280Z 
2023-01-11T21:02:01.5701355Z Running tests...
2023-01-11T21:02:01.5701639Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5701985Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5702222Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18691
2023-01-11T21:02:01.5702381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18692
2023-01-11T21:02:01.5702652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5702770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5703074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5703212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5703363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5703631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5703752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5704033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5704165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5704322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5704424Z skip: Need at least 2 CUDA devices (1.877s)
2023-01-11T21:02:01.5704429Z 
2023-01-11T21:02:01.5704625Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5704690Z Ran 1 test in 1.877s
2023-01-11T21:02:01.5704708Z 
2023-01-11T21:02:01.5704767Z OK (skipped=1)
2023-01-11T21:02:01.5704771Z 
2023-01-11T21:02:01.5704856Z Generating XML reports...
2023-01-11T21:02:01.5705205Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205952.xml
2023-01-11T21:02:01.5705436Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5705712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5705836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5706118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5706254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5706263Z 
2023-01-11T21:02:01.5706324Z Running tests...
2023-01-11T21:02:01.5706521Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5706710Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5706925Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18724
2023-01-11T21:02:01.5707084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18725
2023-01-11T21:02:01.5707360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5707483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5707760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5707897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5708085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5708356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5708479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5708758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5708891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5709054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5709158Z skip: Need at least 2 CUDA devices (1.764s)
2023-01-11T21:02:01.5709163Z 
2023-01-11T21:02:01.5709359Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5709421Z Ran 1 test in 1.764s
2023-01-11T21:02:01.5709456Z 
2023-01-11T21:02:01.5709532Z OK (skipped=1)
2023-01-11T21:02:01.5709537Z 
2023-01-11T21:02:01.5709626Z Generating XML reports...
2023-01-11T21:02:01.5709971Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205955.xml
2023-01-11T21:02:01.5710202Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5710475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5710599Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5710883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5711021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5711026Z 
2023-01-11T21:02:01.5711085Z Running tests...
2023-01-11T21:02:01.5711287Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5711475Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5711689Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18757
2023-01-11T21:02:01.5711847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18758
2023-01-11T21:02:01.5712122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5712247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5712526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5712647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5712814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5713088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5713210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5713486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5713620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5713778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5713885Z skip: Need at least 2 CUDA devices (1.865s)
2023-01-11T21:02:01.5713890Z 
2023-01-11T21:02:01.5714085Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5714147Z Ran 1 test in 1.865s
2023-01-11T21:02:01.5714152Z 
2023-01-11T21:02:01.5714225Z OK (skipped=1)
2023-01-11T21:02:01.5714230Z 
2023-01-11T21:02:01.5714348Z Generating XML reports...
2023-01-11T21:02:01.5714693Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205959.xml
2023-01-11T21:02:01.5714920Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5715190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5715316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5715598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5715720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5715739Z 
2023-01-11T21:02:01.5715799Z Running tests...
2023-01-11T21:02:01.5715990Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5716239Z   test_ddp_comm_hook_allreduce_hook_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5716247Z 
2023-01-11T21:02:01.5716439Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5716516Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5716521Z 
2023-01-11T21:02:01.5716593Z OK (skipped=1)
2023-01-11T21:02:01.5716598Z 
2023-01-11T21:02:01.5716679Z Generating XML reports...
2023-01-11T21:02:01.5717023Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210002.xml
2023-01-11T21:02:01.5717239Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5717508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5717631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5717910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5718044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5718049Z 
2023-01-11T21:02:01.5718119Z Running tests...
2023-01-11T21:02:01.5718312Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5718544Z   test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5718550Z 
2023-01-11T21:02:01.5718740Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5718803Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5718819Z 
2023-01-11T21:02:01.5718875Z OK (skipped=1)
2023-01-11T21:02:01.5718880Z 
2023-01-11T21:02:01.5718962Z Generating XML reports...
2023-01-11T21:02:01.5719307Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210004.xml
2023-01-11T21:02:01.5719537Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5719811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5719934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5720211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5720346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5720351Z 
2023-01-11T21:02:01.5720412Z Running tests...
2023-01-11T21:02:01.5720601Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5720835Z   test_ddp_comm_hook_allreduce_hook_nccl_static_graph (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5720871Z 
2023-01-11T21:02:01.5721068Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5721142Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5721146Z 
2023-01-11T21:02:01.5721217Z OK (skipped=1)
2023-01-11T21:02:01.5721222Z 
2023-01-11T21:02:01.5721304Z Generating XML reports...
2023-01-11T21:02:01.5721648Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210005.xml
2023-01-11T21:02:01.5721875Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5722134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5722256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5722532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5722693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5722701Z 
2023-01-11T21:02:01.5722777Z Running tests...
2023-01-11T21:02:01.5722969Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5723138Z   test_ddp_comm_hook_allreduce_with_then_hook_nccl (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5723355Z This unit test verifies whether a DDP communication hook that calls allreduce and then ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5723361Z 
2023-01-11T21:02:01.5723552Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5723613Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5723618Z 
2023-01-11T21:02:01.5723689Z OK (skipped=1)
2023-01-11T21:02:01.5723693Z 
2023-01-11T21:02:01.5723776Z Generating XML reports...
2023-01-11T21:02:01.5724118Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210007.xml
2023-01-11T21:02:01.5724349Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5724620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5724742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5725020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5725143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5725161Z 
2023-01-11T21:02:01.5725220Z Running tests...
2023-01-11T21:02:01.5725410Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5725573Z   test_ddp_comm_hook_future_passing_gpu_nccl (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5725798Z This unit test verifies whether the Future object is passed properly using nccl backend. ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5725805Z 
2023-01-11T21:02:01.5725997Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5726071Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5726076Z 
2023-01-11T21:02:01.5726147Z OK (skipped=1)
2023-01-11T21:02:01.5726151Z 
2023-01-11T21:02:01.5726234Z Generating XML reports...
2023-01-11T21:02:01.5726563Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210008.xml
2023-01-11T21:02:01.5726790Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5727061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5727183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5727462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5727623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5727629Z 
2023-01-11T21:02:01.5727700Z Running tests...
2023-01-11T21:02:01.5727894Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5728114Z   test_ddp_multi_device_module_config (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5728119Z 
2023-01-11T21:02:01.5728299Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5728374Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5728379Z 
2023-01-11T21:02:01.5728449Z OK (skipped=1)
2023-01-11T21:02:01.5728453Z 
2023-01-11T21:02:01.5728536Z Generating XML reports...
2023-01-11T21:02:01.5728880Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210009.xml
2023-01-11T21:02:01.5729161Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5729434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5729557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5729836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5729958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5729963Z 
2023-01-11T21:02:01.5730034Z Running tests...
2023-01-11T21:02:01.5730229Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5730375Z   test_ddp_packed_sequence (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5730573Z Tests that DDP with ``device_ids`` specified can run a forward and ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:02:01.5730578Z 
2023-01-11T21:02:01.5730776Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5730993Z Ran 1 test in 0.002s
2023-01-11T21:02:01.5730999Z 
2023-01-11T21:02:01.5731071Z OK (skipped=1)
2023-01-11T21:02:01.5731075Z 
2023-01-11T21:02:01.5731158Z Generating XML reports...
2023-01-11T21:02:01.5731491Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210011.xml
2023-01-11T21:02:01.5731723Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5731996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5732119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5732402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5732540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5732547Z 
2023-01-11T21:02:01.5732620Z Running tests...
2023-01-11T21:02:01.5732814Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5733011Z   test_ddp_weight_sharing (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5733028Z 
2023-01-11T21:02:01.5733208Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5733281Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5733286Z 
2023-01-11T21:02:01.5733358Z OK (skipped=1)
2023-01-11T21:02:01.5733366Z 
2023-01-11T21:02:01.5733449Z Generating XML reports...
2023-01-11T21:02:01.5733793Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210012.xml
2023-01-11T21:02:01.5734021Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5734295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5734464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5734730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5734866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5734871Z 
2023-01-11T21:02:01.5734943Z Running tests...
2023-01-11T21:02:01.5735136Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5735352Z   test_ddp_with_lazy_parameters (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5735358Z 
2023-01-11T21:02:01.5735548Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5735623Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5735628Z 
2023-01-11T21:02:01.5735736Z OK (skipped=1)
2023-01-11T21:02:01.5735742Z 
2023-01-11T21:02:01.5735825Z Generating XML reports...
2023-01-11T21:02:01.5736159Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210014.xml
2023-01-11T21:02:01.5736388Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5736662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5736782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5737059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5737195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5737199Z 
2023-01-11T21:02:01.5737270Z Running tests...
2023-01-11T21:02:01.5737462Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5737668Z   test_default_ddp_comm_hooks_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5737688Z 
2023-01-11T21:02:01.5737867Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5737941Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5737946Z 
2023-01-11T21:02:01.5738018Z OK (skipped=1)
2023-01-11T21:02:01.5738022Z 
2023-01-11T21:02:01.5738175Z Generating XML reports...
2023-01-11T21:02:01.5738527Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210015.xml
2023-01-11T21:02:01.5738758Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5739032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5739155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5739426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5739560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5739565Z 
2023-01-11T21:02:01.5739639Z Running tests...
2023-01-11T21:02:01.5739833Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5740055Z   test_default_ddp_comm_hooks_nccl_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5740060Z 
2023-01-11T21:02:01.5740254Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5740330Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5740335Z 
2023-01-11T21:02:01.5740406Z OK (skipped=1)
2023-01-11T21:02:01.5740411Z 
2023-01-11T21:02:01.5740494Z Generating XML reports...
2023-01-11T21:02:01.5740828Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210017.xml
2023-01-11T21:02:01.5741093Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5741367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5741493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5741772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5741905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5741910Z 
2023-01-11T21:02:01.5741982Z Running tests...
2023-01-11T21:02:01.5742179Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5742382Z   test_failure_recovery (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:02:01.5742418Z 
2023-01-11T21:02:01.5742598Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5742672Z Ran 1 test in 0.002s
2023-01-11T21:02:01.5742678Z 
2023-01-11T21:02:01.5742748Z OK (skipped=1)
2023-01-11T21:02:01.5742752Z 
2023-01-11T21:02:01.5742837Z Generating XML reports...
2023-01-11T21:02:01.5743181Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210018.xml
2023-01-11T21:02:01.5743408Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5743680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5743801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5744078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5744204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5744209Z 
2023-01-11T21:02:01.5744280Z Running tests...
2023-01-11T21:02:01.5744477Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5744708Z   test_find_unused_parameters_kwarg_debug_detail (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5744713Z 
2023-01-11T21:02:01.5744905Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5744978Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5744984Z 
2023-01-11T21:02:01.5745054Z OK (skipped=1)
2023-01-11T21:02:01.5745058Z 
2023-01-11T21:02:01.5745145Z Generating XML reports...
2023-01-11T21:02:01.5745483Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210019.xml
2023-01-11T21:02:01.5745701Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5745974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5746097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5746378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5746512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5746516Z 
2023-01-11T21:02:01.5746587Z Running tests...
2023-01-11T21:02:01.5746780Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5747007Z   test_find_unused_parameters_kwarg_debug_info (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5747012Z 
2023-01-11T21:02:01.5747200Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5747262Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5747299Z 
2023-01-11T21:02:01.5747370Z OK (skipped=1)
2023-01-11T21:02:01.5747374Z 
2023-01-11T21:02:01.5747458Z Generating XML reports...
2023-01-11T21:02:01.5747803Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210021.xml
2023-01-11T21:02:01.5748030Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5748300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5748422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5748699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5748850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5748866Z 
2023-01-11T21:02:01.5748926Z Running tests...
2023-01-11T21:02:01.5749153Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5749386Z   test_find_unused_parameters_kwarg_debug_off (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5749391Z 
2023-01-11T21:02:01.5749585Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5749658Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5749663Z 
2023-01-11T21:02:01.5749732Z OK (skipped=1)
2023-01-11T21:02:01.5749737Z 
2023-01-11T21:02:01.5749820Z Generating XML reports...
2023-01-11T21:02:01.5750162Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210022.xml
2023-01-11T21:02:01.5750379Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5750649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5750777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5751055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5751190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5751195Z 
2023-01-11T21:02:01.5751265Z Running tests...
2023-01-11T21:02:01.5751459Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5751704Z   test_find_unused_parameters_kwarg_grad_is_view_debug_detail (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5751709Z 
2023-01-11T21:02:01.5751897Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5751959Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5751964Z 
2023-01-11T21:02:01.5752033Z OK (skipped=1)
2023-01-11T21:02:01.5752038Z 
2023-01-11T21:02:01.5752127Z Generating XML reports...
2023-01-11T21:02:01.5752471Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210024.xml
2023-01-11T21:02:01.5752699Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5752970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5753093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5753370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5753506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5753511Z 
2023-01-11T21:02:01.5753571Z Running tests...
2023-01-11T21:02:01.5753764Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5754004Z   test_find_unused_parameters_kwarg_grad_is_view_debug_info (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5754039Z 
2023-01-11T21:02:01.5754236Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5754315Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5754320Z 
2023-01-11T21:02:01.5754391Z OK (skipped=1)
2023-01-11T21:02:01.5754395Z 
2023-01-11T21:02:01.5754478Z Generating XML reports...
2023-01-11T21:02:01.5754819Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210025.xml
2023-01-11T21:02:01.5755045Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5755303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5755424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5755731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5755869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5755874Z 
2023-01-11T21:02:01.5755948Z Running tests...
2023-01-11T21:02:01.5756139Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5756380Z   test_find_unused_parameters_kwarg_grad_is_view_debug_off (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5756386Z 
2023-01-11T21:02:01.5756578Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5756651Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5756656Z 
2023-01-11T21:02:01.5756716Z OK (skipped=1)
2023-01-11T21:02:01.5756721Z 
2023-01-11T21:02:01.5756806Z Generating XML reports...
2023-01-11T21:02:01.5757152Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210027.xml
2023-01-11T21:02:01.5757384Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5757662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5757788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5758072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5758209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5758214Z 
2023-01-11T21:02:01.5758275Z Running tests...
2023-01-11T21:02:01.5758472Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5758666Z   test_fp16 (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5758671Z 
2023-01-11T21:02:01.5758870Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5758948Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5758953Z 
2023-01-11T21:02:01.5759027Z OK (skipped=1)
2023-01-11T21:02:01.5759031Z 
2023-01-11T21:02:01.5759118Z Generating XML reports...
2023-01-11T21:02:01.5759463Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210028.xml
2023-01-11T21:02:01.5759695Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5759957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5760087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5760370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5760508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5760543Z 
2023-01-11T21:02:01.5760618Z Running tests...
2023-01-11T21:02:01.5760813Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5761036Z   test_fp16_compress_wrapper_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5761042Z 
2023-01-11T21:02:01.5761234Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5761312Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5761316Z 
2023-01-11T21:02:01.5761376Z OK (skipped=1)
2023-01-11T21:02:01.5761381Z 
2023-01-11T21:02:01.5761467Z Generating XML reports...
2023-01-11T21:02:01.5761816Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210030.xml
2023-01-11T21:02:01.5762047Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5762348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5762477Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5762753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5762888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5762893Z 
2023-01-11T21:02:01.5762965Z Running tests...
2023-01-11T21:02:01.5763145Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5763364Z   test_fp16_compress_wrapper_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5763370Z 
2023-01-11T21:02:01.5763561Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5763636Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5763641Z 
2023-01-11T21:02:01.5763718Z OK (skipped=1)
2023-01-11T21:02:01.5763724Z 
2023-01-11T21:02:01.5763811Z Generating XML reports...
2023-01-11T21:02:01.5764150Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210031.xml
2023-01-11T21:02:01.5764381Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5764652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5764762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5765044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5765178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5765182Z 
2023-01-11T21:02:01.5765254Z Running tests...
2023-01-11T21:02:01.5765446Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5765652Z   test_fp16_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5765657Z 
2023-01-11T21:02:01.5765851Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5765928Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5765932Z 
2023-01-11T21:02:01.5765990Z OK (skipped=1)
2023-01-11T21:02:01.5766007Z 
2023-01-11T21:02:01.5766077Z Generating XML reports...
2023-01-11T21:02:01.5766419Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210032.xml
2023-01-11T21:02:01.5766647Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5766918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5767039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5767357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5767492Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5767497Z 
2023-01-11T21:02:01.5767568Z Running tests...
2023-01-11T21:02:01.5767749Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5767988Z   test_grad_layout_1devicemodule_1replicaperprocess (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5767993Z 
2023-01-11T21:02:01.5768186Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5768258Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5768263Z 
2023-01-11T21:02:01.5768333Z OK (skipped=1)
2023-01-11T21:02:01.5768337Z 
2023-01-11T21:02:01.5768419Z Generating XML reports...
2023-01-11T21:02:01.5768802Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210034.xml
2023-01-11T21:02:01.5769034Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5769308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5769419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5769692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5769825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5769830Z 
2023-01-11T21:02:01.5769900Z Running tests...
2023-01-11T21:02:01.5770091Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5770303Z   test_grad_layout_2devicemodule (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5770312Z 
2023-01-11T21:02:01.5770502Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5770572Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5770577Z 
2023-01-11T21:02:01.5770644Z OK (skipped=1)
2023-01-11T21:02:01.5770649Z 
2023-01-11T21:02:01.5770719Z Generating XML reports...
2023-01-11T21:02:01.5771166Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210035.xml
2023-01-11T21:02:01.5771394Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5771665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5771787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5772062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5772200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5772206Z 
2023-01-11T21:02:01.5772278Z Running tests...
2023-01-11T21:02:01.5772457Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5772697Z   test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19030
2023-01-11T21:02:01.5772858Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19031
2023-01-11T21:02:01.5773130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5773252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5773527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5773660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5773881Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5774294Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:02:01.5774953Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:02:01.5775705Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:02:01.5776407Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:02:01.5777124Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:02:01.5777850Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:02:01.5778423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5778612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5779110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5779337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5779624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5780372Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:02:01.5781105Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:02:01.5781792Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:02:01.5782228Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:02:01.5782707Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:02:01.5783145Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:02:01.5783212Z ok (1.862s)
2023-01-11T21:02:01.5783219Z 
2023-01-11T21:02:01.5783442Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5783517Z Ran 1 test in 1.862s
2023-01-11T21:02:01.5783523Z 
2023-01-11T21:02:01.5783582Z OK
2023-01-11T21:02:01.5783586Z 
2023-01-11T21:02:01.5783670Z Generating XML reports...
2023-01-11T21:02:01.5784014Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210037.xml
2023-01-11T21:02:01.5784244Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5784517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5784641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5784926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5785064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5785070Z 
2023-01-11T21:02:01.5785139Z Running tests...
2023-01-11T21:02:01.5785334Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5785548Z   test_multiple_outputs_multiple_backward (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5785562Z 
2023-01-11T21:02:01.5785738Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5785816Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5785821Z 
2023-01-11T21:02:01.5785891Z OK (skipped=1)
2023-01-11T21:02:01.5785896Z 
2023-01-11T21:02:01.5785976Z Generating XML reports...
2023-01-11T21:02:01.5786318Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210040.xml
2023-01-11T21:02:01.5786547Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5786813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5786935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5787205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5787336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5787341Z 
2023-01-11T21:02:01.5787411Z Running tests...
2023-01-11T21:02:01.5787601Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5787837Z   test_multiple_outputs_multiple_backward_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5787843Z 
2023-01-11T21:02:01.5788038Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5788141Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5788146Z 
2023-01-11T21:02:01.5788215Z OK (skipped=1)
2023-01-11T21:02:01.5788220Z 
2023-01-11T21:02:01.5788303Z Generating XML reports...
2023-01-11T21:02:01.5788634Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210042.xml
2023-01-11T21:02:01.5788862Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5789126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5789251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5789530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5789661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5789668Z 
2023-01-11T21:02:01.5789767Z Running tests...
2023-01-11T21:02:01.5789960Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5790193Z   test_nccl_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5790198Z 
2023-01-11T21:02:01.5790376Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5790449Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5790454Z 
2023-01-11T21:02:01.5790524Z OK (skipped=1)
2023-01-11T21:02:01.5790528Z 
2023-01-11T21:02:01.5790612Z Generating XML reports...
2023-01-11T21:02:01.5790958Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210043.xml
2023-01-11T21:02:01.5791183Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5791452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5791574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5791851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5791972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5791977Z 
2023-01-11T21:02:01.5792044Z Running tests...
2023-01-11T21:02:01.5792238Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5792470Z   test_nccl_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5792475Z 
2023-01-11T21:02:01.5792664Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5792736Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5792741Z 
2023-01-11T21:02:01.5792810Z OK (skipped=1)
2023-01-11T21:02:01.5792815Z 
2023-01-11T21:02:01.5792895Z Generating XML reports...
2023-01-11T21:02:01.5793237Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210044.xml
2023-01-11T21:02:01.5793451Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5793718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5793836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5794108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5794237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5794242Z 
2023-01-11T21:02:01.5794308Z Running tests...
2023-01-11T21:02:01.5794501Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5794737Z   test_nccl_backend_2gpu_module (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5794743Z 
2023-01-11T21:02:01.5794932Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5794993Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5794998Z 
2023-01-11T21:02:01.5795064Z OK (skipped=1)
2023-01-11T21:02:01.5795069Z 
2023-01-11T21:02:01.5795147Z Generating XML reports...
2023-01-11T21:02:01.5795488Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210046.xml
2023-01-11T21:02:01.5795714Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5795981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5796099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5796405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5796529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5796545Z 
2023-01-11T21:02:01.5796604Z Running tests...
2023-01-11T21:02:01.5796798Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5797009Z   test_nccl_backend_4gpu_module (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5797014Z 
2023-01-11T21:02:01.5797202Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5797279Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5797284Z 
2023-01-11T21:02:01.5797352Z OK (skipped=1)
2023-01-11T21:02:01.5797357Z 
2023-01-11T21:02:01.5797442Z Generating XML reports...
2023-01-11T21:02:01.5797791Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210047.xml
2023-01-11T21:02:01.5798005Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5798275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5798398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5798673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5798802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5798807Z 
2023-01-11T21:02:01.5798878Z Running tests...
2023-01-11T21:02:01.5799068Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5799296Z   test_nccl_backend_multi_device_ids_not_allowed (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5799307Z 
2023-01-11T21:02:01.5799494Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5799556Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5799568Z 
2023-01-11T21:02:01.5799626Z OK (skipped=1)
2023-01-11T21:02:01.5799630Z 
2023-01-11T21:02:01.5799711Z Generating XML reports...
2023-01-11T21:02:01.5800049Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210049.xml
2023-01-11T21:02:01.5800276Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5800549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5800673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5800945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5801110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5801114Z 
2023-01-11T21:02:01.5801174Z Running tests...
2023-01-11T21:02:01.5801361Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5801596Z   test_nccl_backend_multi_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5801601Z 
2023-01-11T21:02:01.5801788Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5801862Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5801867Z 
2023-01-11T21:02:01.5801934Z OK (skipped=1)
2023-01-11T21:02:01.5801938Z 
2023-01-11T21:02:01.5802021Z Generating XML reports...
2023-01-11T21:02:01.5802361Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210050.xml
2023-01-11T21:02:01.5802623Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5802888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5803005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5803277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5803406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5803411Z 
2023-01-11T21:02:01.5803479Z Running tests...
2023-01-11T21:02:01.5803666Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5803900Z   test_nccl_backend_single_device_module_device_ids_None (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5803905Z 
2023-01-11T21:02:01.5804092Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5804166Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5804171Z 
2023-01-11T21:02:01.5804228Z OK (skipped=1)
2023-01-11T21:02:01.5804232Z 
2023-01-11T21:02:01.5804311Z Generating XML reports...
2023-01-11T21:02:01.5804651Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210052.xml
2023-01-11T21:02:01.5804876Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5805144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5805266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5805546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5805678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5805683Z 
2023-01-11T21:02:01.5805758Z Running tests...
2023-01-11T21:02:01.5805940Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5806176Z   test_nccl_backend_single_device_module_empty_device_ids (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5806181Z 
2023-01-11T21:02:01.5806370Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5806444Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5806449Z 
2023-01-11T21:02:01.5806519Z OK (skipped=1)
2023-01-11T21:02:01.5806525Z 
2023-01-11T21:02:01.5806610Z Generating XML reports...
2023-01-11T21:02:01.5806951Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210053.xml
2023-01-11T21:02:01.5807178Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5807441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5807591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5807868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5807998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5808003Z 
2023-01-11T21:02:01.5808074Z Running tests...
2023-01-11T21:02:01.5808266Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5808485Z   test_nccl_propagate_error_reason (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5808490Z 
2023-01-11T21:02:01.5808679Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5808753Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5808758Z 
2023-01-11T21:02:01.5808816Z OK (skipped=1)
2023-01-11T21:02:01.5808832Z 
2023-01-11T21:02:01.5808931Z Generating XML reports...
2023-01-11T21:02:01.5809274Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210055.xml
2023-01-11T21:02:01.5809501Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5809771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5809899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5810176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5810310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5810315Z 
2023-01-11T21:02:01.5810386Z Running tests...
2023-01-11T21:02:01.5810561Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5810692Z   test_no_grad (__main__.DistributedDataParallelTest)
2023-01-11T21:02:01.5811044Z Note: this test can be sped up by only running it on a CPU module ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5811052Z 
2023-01-11T21:02:01.5811252Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5811328Z Ran 1 test in 0.002s
2023-01-11T21:02:01.5811333Z 
2023-01-11T21:02:01.5811404Z OK (skipped=1)
2023-01-11T21:02:01.5811408Z 
2023-01-11T21:02:01.5811493Z Generating XML reports...
2023-01-11T21:02:01.5811839Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210056.xml
2023-01-11T21:02:01.5812069Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5812325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5812451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5812730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5812869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5812873Z 
2023-01-11T21:02:01.5812946Z Running tests...
2023-01-11T21:02:01.5813138Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5813356Z   test_param_layout_mismatch_error (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5813361Z 
2023-01-11T21:02:01.5813552Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5813614Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5813633Z 
2023-01-11T21:02:01.5813691Z OK (skipped=1)
2023-01-11T21:02:01.5813695Z 
2023-01-11T21:02:01.5813780Z Generating XML reports...
2023-01-11T21:02:01.5814126Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210058.xml
2023-01-11T21:02:01.5814424Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5814697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5814820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5815102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5815238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5815243Z 
2023-01-11T21:02:01.5815303Z Running tests...
2023-01-11T21:02:01.5815497Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5815700Z   test_pass_default_pg (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5815743Z 
2023-01-11T21:02:01.5815936Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5816010Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5816015Z 
2023-01-11T21:02:01.5816086Z OK (skipped=1)
2023-01-11T21:02:01.5816091Z 
2023-01-11T21:02:01.5816175Z Generating XML reports...
2023-01-11T21:02:01.5816521Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210059.xml
2023-01-11T21:02:01.5816752Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5817011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5817136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5817415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5817555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5817559Z 
2023-01-11T21:02:01.5817635Z Running tests...
2023-01-11T21:02:01.5817827Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5818043Z   test_powerSGD_ddp_comm_hook_nccl (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5818048Z 
2023-01-11T21:02:01.5818318Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5818396Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5818401Z 
2023-01-11T21:02:01.5818459Z OK (skipped=1)
2023-01-11T21:02:01.5818464Z 
2023-01-11T21:02:01.5818547Z Generating XML reports...
2023-01-11T21:02:01.5818893Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210100.xml
2023-01-11T21:02:01.5819120Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5819397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5819520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5819798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5819934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5819938Z 
2023-01-11T21:02:01.5819997Z Running tests...
2023-01-11T21:02:01.5820189Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5820416Z   test_powerSGD_ddp_comm_hook_nccl_grad_is_view (__main__.DistributedDataParallelTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5820422Z 
2023-01-11T21:02:01.5820615Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5820690Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5820734Z 
2023-01-11T21:02:01.5820808Z OK (skipped=1)
2023-01-11T21:02:01.5820812Z 
2023-01-11T21:02:01.5820897Z Generating XML reports...
2023-01-11T21:02:01.5821244Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210102.xml
2023-01-11T21:02:01.5821475Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5821735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5821860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5822139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5822276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5822281Z 
2023-01-11T21:02:01.5822354Z Running tests...
2023-01-11T21:02:01.5822602Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5822851Z   test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19223
2023-01-11T21:02:01.5823009Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19224
2023-01-11T21:02:01.5823283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5823394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5823674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5823812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5823982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5824252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5824379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5824660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5824794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5824965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5825061Z skip: Need at least 2 CUDA devices (1.869s)
2023-01-11T21:02:01.5825067Z 
2023-01-11T21:02:01.5825265Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5825342Z Ran 1 test in 1.870s
2023-01-11T21:02:01.5825347Z 
2023-01-11T21:02:01.5825420Z OK (skipped=1)
2023-01-11T21:02:01.5825425Z 
2023-01-11T21:02:01.5825513Z Generating XML reports...
2023-01-11T21:02:01.5825863Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210103.xml
2023-01-11T21:02:01.5826095Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5826369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5826485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5826767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5826902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5826907Z 
2023-01-11T21:02:01.5826985Z Running tests...
2023-01-11T21:02:01.5827178Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5827434Z   test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19256
2023-01-11T21:02:01.5827620Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19257
2023-01-11T21:02:01.5827890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5828013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5828281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5828412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5828573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:02:01.5828844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5828965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5829272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5829406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5829566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:02:01.5829658Z skip: Need at least 2 CUDA devices (1.767s)
2023-01-11T21:02:01.5829674Z 
2023-01-11T21:02:01.5829857Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5829931Z Ran 1 test in 1.767s
2023-01-11T21:02:01.5829936Z 
2023-01-11T21:02:01.5830008Z OK (skipped=1)
2023-01-11T21:02:01.5830012Z 
2023-01-11T21:02:01.5830095Z Generating XML reports...
2023-01-11T21:02:01.5830441Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210107.xml
2023-01-11T21:02:01.5830672Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5830942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5831064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5831330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5831465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5831469Z 
2023-01-11T21:02:01.5831543Z Running tests...
2023-01-11T21:02:01.5831738Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5831944Z   test_invalid_nccl_blocking_wait_env (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5831949Z 
2023-01-11T21:02:01.5832138Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5832213Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5832222Z 
2023-01-11T21:02:01.5832292Z OK (skipped=1)
2023-01-11T21:02:01.5832296Z 
2023-01-11T21:02:01.5832378Z Generating XML reports...
2023-01-11T21:02:01.5832690Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111210110.xml
2023-01-11T21:02:01.5832919Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5833193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5833315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5833594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5833727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5833731Z 
2023-01-11T21:02:01.5833803Z Running tests...
2023-01-11T21:02:01.5834029Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5834218Z   test_nccl_blocking_wait_with_barrier (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5834235Z 
2023-01-11T21:02:01.5834414Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5834488Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5834493Z 
2023-01-11T21:02:01.5834565Z OK (skipped=1)
2023-01-11T21:02:01.5834569Z 
2023-01-11T21:02:01.5834652Z Generating XML reports...
2023-01-11T21:02:01.5834974Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111210112.xml
2023-01-11T21:02:01.5835204Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5835476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5835625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5835895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5836026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5836031Z 
2023-01-11T21:02:01.5836104Z Running tests...
2023-01-11T21:02:01.5836299Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5836500Z   test_nccl_errors_blocking_abort (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5836505Z 
2023-01-11T21:02:01.5836692Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5836766Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5836771Z 
2023-01-11T21:02:01.5836840Z OK (skipped=1)
2023-01-11T21:02:01.5836844Z 
2023-01-11T21:02:01.5836925Z Generating XML reports...
2023-01-11T21:02:01.5837237Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111210113.xml
2023-01-11T21:02:01.5837467Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5837740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5837863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5838145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5838278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5838283Z 
2023-01-11T21:02:01.5838355Z Running tests...
2023-01-11T21:02:01.5838547Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5838752Z   test_nccl_errors_blocking_clean_exit (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5838760Z 
2023-01-11T21:02:01.5838938Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5839012Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5839016Z 
2023-01-11T21:02:01.5839086Z OK (skipped=1)
2023-01-11T21:02:01.5839091Z 
2023-01-11T21:02:01.5839174Z Generating XML reports...
2023-01-11T21:02:01.5839493Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111210114.xml
2023-01-11T21:02:01.5839721Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5839992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5840114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5840390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5840543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5840548Z 
2023-01-11T21:02:01.5840618Z Running tests...
2023-01-11T21:02:01.5840810Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5841021Z   test_nccl_errors_blocking_nonzero_exit (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5841026Z 
2023-01-11T21:02:01.5841216Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5841290Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5841294Z 
2023-01-11T21:02:01.5841365Z OK (skipped=1)
2023-01-11T21:02:01.5841370Z 
2023-01-11T21:02:01.5841452Z Generating XML reports...
2023-01-11T21:02:01.5841763Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111210116.xml
2023-01-11T21:02:01.5842020Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5842293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5842416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5842698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5842832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5842837Z 
2023-01-11T21:02:01.5842909Z Running tests...
2023-01-11T21:02:01.5843104Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5843305Z   test_nccl_errors_blocking_sigkill (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5843310Z 
2023-01-11T21:02:01.5843487Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5843561Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5843570Z 
2023-01-11T21:02:01.5843640Z OK (skipped=1)
2023-01-11T21:02:01.5843644Z 
2023-01-11T21:02:01.5843728Z Generating XML reports...
2023-01-11T21:02:01.5844051Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111210117.xml
2023-01-11T21:02:01.5844277Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5844547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5844670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5844947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5845069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5845073Z 
2023-01-11T21:02:01.5845144Z Running tests...
2023-01-11T21:02:01.5845341Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5845541Z   test_nccl_errors_blocking_sigterm (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5845547Z 
2023-01-11T21:02:01.5845739Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5845814Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5845819Z 
2023-01-11T21:02:01.5845890Z OK (skipped=1)
2023-01-11T21:02:01.5845895Z 
2023-01-11T21:02:01.5845976Z Generating XML reports...
2023-01-11T21:02:01.5846297Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111210119.xml
2023-01-11T21:02:01.5846511Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5846781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5846903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5847212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5847349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5847354Z 
2023-01-11T21:02:01.5847426Z Running tests...
2023-01-11T21:02:01.5847617Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5847812Z   test_nccl_errors_nonblocking (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5847816Z 
2023-01-11T21:02:01.5848007Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5848069Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5848074Z 
2023-01-11T21:02:01.5848144Z OK (skipped=1)
2023-01-11T21:02:01.5848148Z 
2023-01-11T21:02:01.5848229Z Generating XML reports...
2023-01-11T21:02:01.5848575Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111210120.xml
2023-01-11T21:02:01.5848808Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5849076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5849202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5849480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5849600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5849617Z 
2023-01-11T21:02:01.5849675Z Running tests...
2023-01-11T21:02:01.5849865Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5850050Z   test_nccl_timeout (__main__.NcclErrorHandlingTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5850059Z 
2023-01-11T21:02:01.5850249Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5850327Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5850333Z 
2023-01-11T21:02:01.5850404Z OK (skipped=1)
2023-01-11T21:02:01.5850408Z 
2023-01-11T21:02:01.5850490Z Generating XML reports...
2023-01-11T21:02:01.5850807Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111210122.xml
2023-01-11T21:02:01.5851187Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5851462Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5851587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5851866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5852004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5852010Z 
2023-01-11T21:02:01.5852085Z Running tests...
2023-01-11T21:02:01.5852278Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5852534Z   test_all_to_all_single (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5852539Z 
2023-01-11T21:02:01.5852730Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5852791Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5852796Z 
2023-01-11T21:02:01.5852867Z OK (skipped=1)
2023-01-11T21:02:01.5852872Z 
2023-01-11T21:02:01.5852955Z Generating XML reports...
2023-01-11T21:02:01.5853378Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111210123.xml
2023-01-11T21:02:01.5853608Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5853943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5854066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5854348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5854471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5854488Z 
2023-01-11T21:02:01.5854547Z Running tests...
2023-01-11T21:02:01.5854744Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5854998Z   test_allgather_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5855004Z 
2023-01-11T21:02:01.5855193Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5855308Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5855314Z 
2023-01-11T21:02:01.5855389Z OK (skipped=1)
2023-01-11T21:02:01.5855393Z 
2023-01-11T21:02:01.5855477Z Generating XML reports...
2023-01-11T21:02:01.5855894Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111210124.xml
2023-01-11T21:02:01.5856111Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5856382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5856503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5856777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5856908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5856913Z 
2023-01-11T21:02:01.5856987Z Running tests...
2023-01-11T21:02:01.5857183Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5857442Z   test_allreduce_coalesced (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5857446Z 
2023-01-11T21:02:01.5857637Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5857698Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5857713Z 
2023-01-11T21:02:01.5857771Z OK (skipped=1)
2023-01-11T21:02:01.5857776Z 
2023-01-11T21:02:01.5857858Z Generating XML reports...
2023-01-11T21:02:01.5858376Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111210126.xml
2023-01-11T21:02:01.5858606Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5858880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5859006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5859284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5859417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5859421Z 
2023-01-11T21:02:01.5859481Z Running tests...
2023-01-11T21:02:01.5859676Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5859928Z   test_collectives (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5859933Z 
2023-01-11T21:02:01.5860126Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5860201Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5860206Z 
2023-01-11T21:02:01.5860277Z OK (skipped=1)
2023-01-11T21:02:01.5860282Z 
2023-01-11T21:02:01.5860406Z Generating XML reports...
2023-01-11T21:02:01.5860828Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111210127.xml
2023-01-11T21:02:01.5861058Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5861316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5861438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5861717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5861851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5861856Z 
2023-01-11T21:02:01.5861928Z Running tests...
2023-01-11T21:02:01.5862121Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5862411Z   test_reduce_scatter_base (__main__.NcclProcessGroupWithDispatchedCollectivesTests) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5862417Z 
2023-01-11T21:02:01.5862609Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5862683Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5862691Z 
2023-01-11T21:02:01.5862750Z OK (skipped=1)
2023-01-11T21:02:01.5862754Z 
2023-01-11T21:02:01.5862836Z Generating XML reports...
2023-01-11T21:02:01.5863248Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111210129.xml
2023-01-11T21:02:01.5863476Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5863746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5863868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5864150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5864283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5864287Z 
2023-01-11T21:02:01.5864359Z Running tests...
2023-01-11T21:02:01.5864539Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5864737Z   test_init_no_gpus (__main__.ProcessGroupNCCLNoGPUTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5864743Z 
2023-01-11T21:02:01.5864930Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5865003Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5865008Z 
2023-01-11T21:02:01.5865078Z OK (skipped=1)
2023-01-11T21:02:01.5865082Z 
2023-01-11T21:02:01.5865164Z Generating XML reports...
2023-01-11T21:02:01.5865511Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20230111210130.xml
2023-01-11T21:02:01.5865739Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5865995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5866117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5866391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5866526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5866531Z 
2023-01-11T21:02:01.5866605Z Running tests...
2023-01-11T21:02:01.5866797Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5866996Z   test_allgather_base_basics (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5867001Z 
2023-01-11T21:02:01.5867235Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5867312Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5867316Z 
2023-01-11T21:02:01.5867375Z OK (skipped=1)
2023-01-11T21:02:01.5867394Z 
2023-01-11T21:02:01.5867464Z Generating XML reports...
2023-01-11T21:02:01.5867782Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210132.xml
2023-01-11T21:02:01.5868010Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5868284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5868407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5868684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5868860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5868867Z 
2023-01-11T21:02:01.5868941Z Running tests...
2023-01-11T21:02:01.5869120Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5869316Z   test_allgather_base_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5869325Z 
2023-01-11T21:02:01.5869516Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5869591Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5869596Z 
2023-01-11T21:02:01.5869667Z OK (skipped=1)
2023-01-11T21:02:01.5869672Z 
2023-01-11T21:02:01.5869757Z Generating XML reports...
2023-01-11T21:02:01.5870079Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210133.xml
2023-01-11T21:02:01.5870309Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5870584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5870698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5870979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5871114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5871119Z 
2023-01-11T21:02:01.5871191Z Running tests...
2023-01-11T21:02:01.5871386Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5871576Z   test_allgather_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5871581Z 
2023-01-11T21:02:01.5871770Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5871847Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5871852Z 
2023-01-11T21:02:01.5871910Z OK (skipped=1)
2023-01-11T21:02:01.5871930Z 
2023-01-11T21:02:01.5872004Z Generating XML reports...
2023-01-11T21:02:01.5872327Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210135.xml
2023-01-11T21:02:01.5872554Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5872826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5872950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5873232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5873367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5873372Z 
2023-01-11T21:02:01.5873447Z Running tests...
2023-01-11T21:02:01.5873630Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5873856Z   test_allreduce_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:02:01.5873861Z 
2023-01-11T21:02:01.5874054Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5874131Z Ran 1 test in 0.003s
2023-01-11T21:02:01.5874136Z 
2023-01-11T21:02:01.5874208Z OK (skipped=1)
2023-01-11T21:02:01.5874212Z 
2023-01-11T21:02:01.5874297Z Generating XML reports...
2023-01-11T21:02:01.5874620Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210136.xml
2023-01-11T21:02:01.5874848Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5875124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5875236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5875541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5875681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5875686Z 
2023-01-11T21:02:01.5875761Z Running tests...
2023-01-11T21:02:01.5875957Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5876137Z   test_barrier (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5876142Z 
2023-01-11T21:02:01.5876330Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5876404Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5876409Z 
2023-01-11T21:02:01.5876470Z OK (skipped=1)
2023-01-11T21:02:01.5876486Z 
2023-01-11T21:02:01.5876555Z Generating XML reports...
2023-01-11T21:02:01.5876872Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210137.xml
2023-01-11T21:02:01.5877102Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5877371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5877495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5877770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5877905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5877910Z 
2023-01-11T21:02:01.5877980Z Running tests...
2023-01-11T21:02:01.5878159Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5878347Z   test_broadcast_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5878352Z 
2023-01-11T21:02:01.5878539Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5878617Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5878622Z 
2023-01-11T21:02:01.5878692Z OK (skipped=1)
2023-01-11T21:02:01.5878696Z 
2023-01-11T21:02:01.5878778Z Generating XML reports...
2023-01-11T21:02:01.5879094Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210139.xml
2023-01-11T21:02:01.5879321Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5879595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5879706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5879983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5880115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5880119Z 
2023-01-11T21:02:01.5880256Z Running tests...
2023-01-11T21:02:01.5880451Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5880642Z   test_empty_tensors (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5880647Z 
2023-01-11T21:02:01.5880838Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5880914Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5880919Z 
2023-01-11T21:02:01.5880992Z OK (skipped=1)
2023-01-11T21:02:01.5880996Z 
2023-01-11T21:02:01.5881067Z Generating XML reports...
2023-01-11T21:02:01.5881387Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210140.xml
2023-01-11T21:02:01.5881616Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5881886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5882044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5882326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5882458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5882463Z 
2023-01-11T21:02:01.5882536Z Running tests...
2023-01-11T21:02:01.5882717Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5882909Z   test_gather_checks (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:02:01.5882915Z 
2023-01-11T21:02:01.5883103Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5883179Z Ran 1 test in 0.002s
2023-01-11T21:02:01.5883183Z 
2023-01-11T21:02:01.5883256Z OK (skipped=1)
2023-01-11T21:02:01.5883260Z 
2023-01-11T21:02:01.5883342Z Generating XML reports...
2023-01-11T21:02:01.5883660Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210142.xml
2023-01-11T21:02:01.5883888Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5884159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5884269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5884544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5884679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5884684Z 
2023-01-11T21:02:01.5884756Z Running tests...
2023-01-11T21:02:01.5884946Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5885132Z   test_gather_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5885142Z 
2023-01-11T21:02:01.5885331Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5885404Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5885409Z 
2023-01-11T21:02:01.5885479Z OK (skipped=1)
2023-01-11T21:02:01.5885484Z 
2023-01-11T21:02:01.5885555Z Generating XML reports...
2023-01-11T21:02:01.5885875Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210143.xml
2023-01-11T21:02:01.5886105Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5886376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5886497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5886774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5886939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5886944Z 
2023-01-11T21:02:01.5887020Z Running tests...
2023-01-11T21:02:01.5887202Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5887391Z   test_gather_stress (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:02:01.5887396Z 
2023-01-11T21:02:01.5887589Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5887663Z Ran 1 test in 0.002s
2023-01-11T21:02:01.5887668Z 
2023-01-11T21:02:01.5887739Z OK (skipped=1)
2023-01-11T21:02:01.5887743Z 
2023-01-11T21:02:01.5887825Z Generating XML reports...
2023-01-11T21:02:01.5888142Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210145.xml
2023-01-11T21:02:01.5888370Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5888669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5888781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5889059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5889193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5889198Z 
2023-01-11T21:02:01.5889272Z Running tests...
2023-01-11T21:02:01.5889462Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5889658Z   test_nccl_dist_backend_error (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5889663Z 
2023-01-11T21:02:01.5889855Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5889931Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5889935Z 
2023-01-11T21:02:01.5890010Z OK (skipped=1)
2023-01-11T21:02:01.5890015Z 
2023-01-11T21:02:01.5890085Z Generating XML reports...
2023-01-11T21:02:01.5890403Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210146.xml
2023-01-11T21:02:01.5890632Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5891057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5891186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5891467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5891603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5891608Z 
2023-01-11T21:02:01.5891681Z Running tests...
2023-01-11T21:02:01.5891867Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5892057Z   test_reduce_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:02:01.5892062Z 
2023-01-11T21:02:01.5892255Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5892332Z Ran 1 test in 0.002s
2023-01-11T21:02:01.5892336Z 
2023-01-11T21:02:01.5892411Z OK (skipped=1)
2023-01-11T21:02:01.5892415Z 
2023-01-11T21:02:01.5892501Z Generating XML reports...
2023-01-11T21:02:01.5892823Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210148.xml
2023-01-11T21:02:01.5893055Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5893332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5893443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5893780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5893915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5893920Z 
2023-01-11T21:02:01.5893995Z Running tests...
2023-01-11T21:02:01.5894192Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5894402Z   test_reduce_scatter_base_basics (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5894407Z 
2023-01-11T21:02:01.5894600Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5894680Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5894685Z 
2023-01-11T21:02:01.5894760Z OK (skipped=1)
2023-01-11T21:02:01.5894764Z 
2023-01-11T21:02:01.5894837Z Generating XML reports...
2023-01-11T21:02:01.5895197Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210149.xml
2023-01-11T21:02:01.5895436Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5895708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5895832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5896105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5896234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5896239Z 
2023-01-11T21:02:01.5896305Z Running tests...
2023-01-11T21:02:01.5896494Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5896677Z   test_reduce_scatter_base_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5896682Z 
2023-01-11T21:02:01.5896877Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5896953Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5896957Z 
2023-01-11T21:02:01.5897028Z OK (skipped=1)
2023-01-11T21:02:01.5897032Z 
2023-01-11T21:02:01.5897116Z Generating XML reports...
2023-01-11T21:02:01.5897431Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210150.xml
2023-01-11T21:02:01.5897659Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5897929Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5898040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5898387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5898523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5898530Z 
2023-01-11T21:02:01.5898602Z Running tests...
2023-01-11T21:02:01.5898796Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5898991Z   test_reduce_scatter_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.003s)
2023-01-11T21:02:01.5898996Z 
2023-01-11T21:02:01.5899189Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5899264Z Ran 1 test in 0.004s
2023-01-11T21:02:01.5899269Z 
2023-01-11T21:02:01.5899340Z OK (skipped=1)
2023-01-11T21:02:01.5899345Z 
2023-01-11T21:02:01.5899416Z Generating XML reports...
2023-01-11T21:02:01.5899738Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210152.xml
2023-01-11T21:02:01.5899967Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5900243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5900406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5900687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5900821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5900826Z 
2023-01-11T21:02:01.5900898Z Running tests...
2023-01-11T21:02:01.5901093Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5901269Z   test_scatter_checks (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5901274Z 
2023-01-11T21:02:01.5901462Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5901536Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5901540Z 
2023-01-11T21:02:01.5901611Z OK (skipped=1)
2023-01-11T21:02:01.5901619Z 
2023-01-11T21:02:01.5901729Z Generating XML reports...
2023-01-11T21:02:01.5902052Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210153.xml
2023-01-11T21:02:01.5902276Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5902549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5902670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5902935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5903068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5903073Z 
2023-01-11T21:02:01.5903144Z Running tests...
2023-01-11T21:02:01.5903335Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5903519Z   test_scatter_ops (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5903527Z 
2023-01-11T21:02:01.5903719Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5903793Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5903798Z 
2023-01-11T21:02:01.5903868Z OK (skipped=1)
2023-01-11T21:02:01.5903873Z 
2023-01-11T21:02:01.5903943Z Generating XML reports...
2023-01-11T21:02:01.5904261Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210155.xml
2023-01-11T21:02:01.5904488Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5904759Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5904881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5905160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5905295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5905300Z 
2023-01-11T21:02:01.5905373Z Running tests...
2023-01-11T21:02:01.5905564Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5905740Z   test_scatter_stress (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5905745Z 
2023-01-11T21:02:01.5905934Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5906007Z Ran 1 test in 0.002s
2023-01-11T21:02:01.5906012Z 
2023-01-11T21:02:01.5906083Z OK (skipped=1)
2023-01-11T21:02:01.5906088Z 
2023-01-11T21:02:01.5906170Z Generating XML reports...
2023-01-11T21:02:01.5906487Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210156.xml
2023-01-11T21:02:01.5906717Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5907017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5907137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5907400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5907533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5907538Z 
2023-01-11T21:02:01.5907609Z Running tests...
2023-01-11T21:02:01.5907799Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5907982Z   test_send_recv (__main__.ProcessGroupNCCLTest) ... skip: c10d was not compiled with the NCCL backend (0.001s)
2023-01-11T21:02:01.5907987Z 
2023-01-11T21:02:01.5908177Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5908287Z Ran 1 test in 0.001s
2023-01-11T21:02:01.5908292Z 
2023-01-11T21:02:01.5908363Z OK (skipped=1)
2023-01-11T21:02:01.5908368Z 
2023-01-11T21:02:01.5908438Z Generating XML reports...
2023-01-11T21:02:01.5908760Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210158.xml
2023-01-11T21:02:01.5908986Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5909258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5909379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5909656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5909790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5909795Z 
2023-01-11T21:02:01.5909871Z Running tests...
2023-01-11T21:02:01.5910067Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5910233Z   test_common_errors (__main__.RendezvousEnvTest) ... skip: c10d was not compiled with the NCCL backend (0.003s)
2023-01-11T21:02:01.5910249Z 
2023-01-11T21:02:01.5910425Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5910497Z Ran 1 test in 0.003s
2023-01-11T21:02:01.5910502Z 
2023-01-11T21:02:01.5910571Z OK (skipped=1)
2023-01-11T21:02:01.5910575Z 
2023-01-11T21:02:01.5910659Z Generating XML reports...
2023-01-11T21:02:01.5910968Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20230111210159.xml
2023-01-11T21:02:01.5911195Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_nccl
2023-01-11T21:02:01.5911466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:02:01.5911592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:02:01.5911855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:02:01.5911988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:02:01.5911992Z 
2023-01-11T21:02:01.5912064Z Running tests...
2023-01-11T21:02:01.5912252Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5912435Z   test_default_store_timeout_nccl (__main__.TimeoutTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:02:01.5912439Z 
2023-01-11T21:02:01.5912630Z ----------------------------------------------------------------------
2023-01-11T21:02:01.5912704Z Ran 1 test in 0.000s
2023-01-11T21:02:01.5912708Z 
2023-01-11T21:02:01.5912780Z OK (skipped=1)
2023-01-11T21:02:01.5912785Z 
2023-01-11T21:02:01.5912867Z Generating XML reports...
2023-01-11T21:02:01.5913147Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20230111210201.xml
2023-01-11T21:02:01.5913187Z 
2023-01-11T21:02:01.5913574Z ##[endgroup]
2023-01-11T21:02:01.5913896Z FINISHED PRINTING LOG FILE of distributed/test_c10d_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_nccl_o6fdxhvm)
2023-01-11T21:02:01.5913901Z 
2023-01-11T21:02:01.5914114Z Running distributed/fsdp/test_fsdp_summon_full_params ... [2023-01-11 21:02:01.531764]
2023-01-11T21:02:01.5914491Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_summon_full_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:02:01.532171]
2023-01-11T21:03:27.1775117Z 
2023-01-11T21:03:27.1775896Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_summon_full_params
2023-01-11T21:03:27.1777282Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_summon_full_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_summon_full_params_t9w56nt6)
2023-01-11T21:03:27.1778399Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params
2023-01-11T21:03:27.1778704Z 
2023-01-11T21:03:27.1778805Z Running tests...
2023-01-11T21:03:27.1780438Z ----------------------------------------------------------------------
2023-01-11T21:03:27.1781321Z   test_cannot_summon_full_params_from_backward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19649
2023-01-11T21:03:27.1781999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19650
2023-01-11T21:03:27.1782885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1783494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1784183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1784779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1785226Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1785830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1786212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1786701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1787110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1787505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1788064Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1788648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1789096Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1789502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1789817Z dist init r=0, world=2
2023-01-11T21:03:27.1789997Z dist init r=1, world=2
2023-01-11T21:03:27.1790218Z skip: Need at least 2 CUDA devices (1.864s)
2023-01-11T21:03:27.1790674Z   test_cannot_summon_full_params_from_forward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19677
2023-01-11T21:03:27.1791289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19678
2023-01-11T21:03:27.1791808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1792421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1792921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1793329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1793711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1794242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1794688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1795274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1795750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1796216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1796841Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1797490Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1797942Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1798440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1798766Z dist init r=0, world=2
2023-01-11T21:03:27.1799049Z dist init r=1, world=2
2023-01-11T21:03:27.1799275Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:03:27.1799569Z   test_named_parameters_buffers_prefix__recurse_False (__main__.TestSummonFullParams)
2023-01-11T21:03:27.1800082Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19705
2023-01-11T21:03:27.1800582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19706
2023-01-11T21:03:27.1801184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1801620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1802179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1802597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1803043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1803630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1804012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1804566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1804985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1805441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1806052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1806674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1807159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1807626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1807951Z dist init r=1, world=2
2023-01-11T21:03:27.1808131Z dist init r=0, world=2
2023-01-11T21:03:27.1808548Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1808898Z   test_named_parameters_buffers_prefix__recurse_True (__main__.TestSummonFullParams)
2023-01-11T21:03:27.1809422Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19733
2023-01-11T21:03:27.1809931Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19734
2023-01-11T21:03:27.1810466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1811071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1811680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1812084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1812717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1813411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1813816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1814345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1814713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1815097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1815609Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1816118Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1816549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1817160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1817593Z dist init r=1, world=2
2023-01-11T21:03:27.1817883Z dist init r=0, world=2
2023-01-11T21:03:27.1818254Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:03:27.1818791Z   test_named_parameters_buffers_prefix_test_prefix_recurse_False (__main__.TestSummonFullParams)
2023-01-11T21:03:27.1819195Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19761
2023-01-11T21:03:27.1819585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19762
2023-01-11T21:03:27.1820049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1820377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1820802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1821143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1821478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1821944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1822258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1822684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1823023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1823340Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1823838Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1824457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1824844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1825173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1825426Z dist init r=1, world=2
2023-01-11T21:03:27.1825606Z dist init r=0, world=2
2023-01-11T21:03:27.1825797Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1826096Z   test_named_parameters_buffers_prefix_test_prefix_recurse_True (__main__.TestSummonFullParams)
2023-01-11T21:03:27.1826508Z Tests that ``named_parameters()`` and ``named_buffers()`` for a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19789
2023-01-11T21:03:27.1826933Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19790
2023-01-11T21:03:27.1827384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1827715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1828141Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1828468Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1828799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1829264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1829589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1830003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1830344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1830677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1831171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1831675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1832056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1832402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1832645Z dist init r=0, world=2
2023-01-11T21:03:27.1832823Z dist init r=1, world=2
2023-01-11T21:03:27.1833026Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1833444Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19817
2023-01-11T21:03:27.1833882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19818
2023-01-11T21:03:27.1834331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1834658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1835073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1835418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1835750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1836214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1836568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1837359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1837697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1838027Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1838505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1839016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1839397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1839777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1840027Z dist init r=1, world=2
2023-01-11T21:03:27.1840207Z dist init r=0, world=2
2023-01-11T21:03:27.1840414Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1840818Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19845
2023-01-11T21:03:27.1841264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19846
2023-01-11T21:03:27.1841715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1842040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1842449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1842791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1843124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1843579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1843901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1844322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1844658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1844980Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1845469Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1861376Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1861880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1862238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1862492Z dist init r=0, world=2
2023-01-11T21:03:27.1862675Z dist init r=1, world=2
2023-01-11T21:03:27.1862883Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1863290Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19873
2023-01-11T21:03:27.1863745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19874
2023-01-11T21:03:27.1864257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1864595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1865158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1865507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1865848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1866303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1866630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1867059Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1867407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1867727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1868273Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1868794Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1869178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1869511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1869765Z dist init r=1, world=2
2023-01-11T21:03:27.1869943Z dist init r=0, world=2
2023-01-11T21:03:27.1870136Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1870553Z   test_params_are_unflattenned_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19901
2023-01-11T21:03:27.1871001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19902
2023-01-11T21:03:27.1871458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1871774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1872201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1872541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1872862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1873327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1873652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1874074Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1874408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1874738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1875232Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1875750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1876120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1876463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1876718Z dist init r=0, world=2
2023-01-11T21:03:27.1876884Z dist init r=1, world=2
2023-01-11T21:03:27.1877089Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1877504Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19929
2023-01-11T21:03:27.1877992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19930
2023-01-11T21:03:27.1878434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1878760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1879186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1879515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1879847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1880349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1880678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1881093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1881438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1881776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1882265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1882774Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1883162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1883505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1883753Z dist init r=0, world=2
2023-01-11T21:03:27.1883934Z dist init r=1, world=2
2023-01-11T21:03:27.1884140Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1884552Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19957
2023-01-11T21:03:27.1884984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19958
2023-01-11T21:03:27.1885438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1885764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1886176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1886517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1886854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1887320Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1887631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1888054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1888394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1888725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1889200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1889715Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1890138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1890471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1890725Z dist init r=0, world=2
2023-01-11T21:03:27.1891100Z dist init r=1, world=2
2023-01-11T21:03:27.1891315Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1891716Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19985
2023-01-11T21:03:27.1892167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19986
2023-01-11T21:03:27.1892622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1892937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1893442Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1893788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1894129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1894584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1894911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1895344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1895686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1896005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1896504Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1897026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1897397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1897740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1897992Z dist init r=0, world=2
2023-01-11T21:03:27.1898172Z dist init r=1, world=2
2023-01-11T21:03:27.1898420Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1898835Z   test_params_are_unflattenned_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20013
2023-01-11T21:03:27.1899283Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20014
2023-01-11T21:03:27.1899729Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1900060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1900489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1900836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1901159Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1901625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1901954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1902384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1902764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1903096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1903588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1904090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1904473Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1904818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1905081Z dist init r=0, world=2
2023-01-11T21:03:27.1905249Z dist init r=1, world=2
2023-01-11T21:03:27.1905460Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1905907Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20041
2023-01-11T21:03:27.1906341Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20042
2023-01-11T21:03:27.1906790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1907117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1907544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1907874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1908208Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1908671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1909000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1909410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1909751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1910081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1910558Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1911075Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1911459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1911805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1912053Z dist init r=0, world=2
2023-01-11T21:03:27.1912233Z dist init r=1, world=2
2023-01-11T21:03:27.1912438Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1912833Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20069
2023-01-11T21:03:27.1913277Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20070
2023-01-11T21:03:27.1913728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1914057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1914465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1914805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1915172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1915635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1915945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1916367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1916706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1917023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1917515Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1918029Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1918444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1918778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1919032Z dist init r=0, world=2
2023-01-11T21:03:27.1919213Z dist init r=1, world=2
2023-01-11T21:03:27.1919406Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1919817Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20097
2023-01-11T21:03:27.1920259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20098
2023-01-11T21:03:27.1920713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1921024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1921452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1921794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1922130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1922586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1922911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1923336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1923662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1923993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1924489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1925001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1925371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1925714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1925970Z dist init r=1, world=2
2023-01-11T21:03:27.1926135Z dist init r=0, world=2
2023-01-11T21:03:27.1926339Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1926749Z   test_params_count_and_value_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20125
2023-01-11T21:03:27.1927197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20126
2023-01-11T21:03:27.1927667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1927989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1928412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1928739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1929069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1929531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1929852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1930261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1930633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1931158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1931652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1932156Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1932540Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1932884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1933129Z dist init r=0, world=2
2023-01-11T21:03:27.1933309Z dist init r=1, world=2
2023-01-11T21:03:27.1933517Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1933917Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20153
2023-01-11T21:03:27.1934363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20154
2023-01-11T21:03:27.1934808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1935131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1935540Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1935881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1936212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1936676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1936994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1937415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1937750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1938068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1938619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1939136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1939519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1939848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1940169Z dist init r=1, world=2
2023-01-11T21:03:27.1940352Z dist init r=0, world=2
2023-01-11T21:03:27.1940547Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1940960Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20181
2023-01-11T21:03:27.1941408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20182
2023-01-11T21:03:27.1941860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1942174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1942597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1942942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1943315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1943770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1944093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1944516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1944841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1945168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1945657Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1946169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1946543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1946884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1947132Z dist init r=1, world=2
2023-01-11T21:03:27.1947312Z dist init r=0, world=2
2023-01-11T21:03:27.1947504Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1947912Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20209
2023-01-11T21:03:27.1948351Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20210
2023-01-11T21:03:27.1948787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1949112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1949532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1949874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1950191Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1950656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1950974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1951380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1951717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1952047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1952537Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1953077Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1953457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1953801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1954061Z dist init r=1, world=2
2023-01-11T21:03:27.1954290Z dist init r=0, world=2
2023-01-11T21:03:27.1954498Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1954912Z   test_params_count_and_value_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20237
2023-01-11T21:03:27.1955339Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20238
2023-01-11T21:03:27.1955820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1956148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1956579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1956916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1957252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1957719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1958026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1958450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1958799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1959134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1959615Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1960137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1960523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1960870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1961115Z dist init r=1, world=2
2023-01-11T21:03:27.1961296Z dist init r=0, world=2
2023-01-11T21:03:27.1961504Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1961761Z   test_raises_rank0_with_writeback (__main__.TestSummonFullParams)
2023-01-11T21:03:27.1962137Z Tests that ``summon_full_params()`` with both ``rank0_only=True`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20265
2023-01-11T21:03:27.1962520Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20266
2023-01-11T21:03:27.1962971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1963285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1963707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1964045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1964362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1964834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1965195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1965614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1965944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1966272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1966759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1967258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1967642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1967987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1968275Z dist init r=1, world=2
2023-01-11T21:03:27.1968445Z dist init r=0, world=2
2023-01-11T21:03:27.1968651Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:03:27.1969085Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20293
2023-01-11T21:03:27.1969628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20294
2023-01-11T21:03:27.1970062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1970388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1970810Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1971316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1971644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1972115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1972439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1972845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1973183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1973514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1974006Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1974506Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1974888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1975229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1975484Z dist init r=0, world=2
2023-01-11T21:03:27.1975648Z dist init r=1, world=2
2023-01-11T21:03:27.1975855Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1976288Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20321
2023-01-11T21:03:27.1976740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20322
2023-01-11T21:03:27.1977188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1977513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1977999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1978404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1978737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1979205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1979531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1979940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1980285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1980614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1981153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1981672Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1982053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1982394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1982638Z dist init r=0, world=2
2023-01-11T21:03:27.1982820Z dist init r=1, world=2
2023-01-11T21:03:27.1983027Z skip: Need at least 2 CUDA devices (1.615s)
2023-01-11T21:03:27.1983448Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20349
2023-01-11T21:03:27.1983919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20350
2023-01-11T21:03:27.1984374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1984706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1985117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1985460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1985790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1986319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1986652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1987072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1987416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1987735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1988222Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1988737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1989119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1989450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1989701Z dist init r=1, world=2
2023-01-11T21:03:27.1989882Z dist init r=0, world=2
2023-01-11T21:03:27.1990075Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.1990507Z   test_reshard_outside_forward_backward_iteration_rank0_only_False_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20377
2023-01-11T21:03:27.1991013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20378
2023-01-11T21:03:27.1991461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1991774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1992201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1992542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1992873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.1993365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1993691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1994116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1994441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1994769Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.1995259Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1995773Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.1996141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.1996486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.1996743Z dist init r=1, world=2
2023-01-11T21:03:27.1996909Z dist init r=0, world=2
2023-01-11T21:03:27.1997114Z skip: Need at least 2 CUDA devices (1.615s)
2023-01-11T21:03:27.1997545Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20405
2023-01-11T21:03:27.1998012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20406
2023-01-11T21:03:27.1998447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.1998771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.1999196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.1999543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.1999863Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2000324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2000644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2001056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2001393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2001721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2002209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2002713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2003136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2003478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2003736Z dist init r=1, world=2
2023-01-11T21:03:27.2003903Z dist init r=0, world=2
2023-01-11T21:03:27.2004104Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.2004535Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20433
2023-01-11T21:03:27.2004987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20434
2023-01-11T21:03:27.2005434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2005794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2006220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2006548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2006879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2007341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2007649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2008072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2008410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2008746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2009231Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2009752Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2010139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2010486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2010730Z dist init r=0, world=2
2023-01-11T21:03:27.2011062Z dist init r=1, world=2
2023-01-11T21:03:27.2011274Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.2011694Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20461
2023-01-11T21:03:27.2012165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20462
2023-01-11T21:03:27.2012618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2012946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2013362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2013706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2014042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2014495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2014820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2015251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2015653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2015975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2016466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2016988Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2017371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2017703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2017957Z dist init r=0, world=2
2023-01-11T21:03:27.2018139Z dist init r=1, world=2
2023-01-11T21:03:27.2018399Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.2018882Z   test_reshard_outside_forward_backward_iteration_rank0_only_True_offload_to_cpu_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20489
2023-01-11T21:03:27.2019347Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20490
2023-01-11T21:03:27.2019802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2020117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2020541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2020885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2021205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2021679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2022006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2022431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2022759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2023088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2023578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2024095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2024464Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2024811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2025069Z dist init r=0, world=2
2023-01-11T21:03:27.2025232Z dist init r=1, world=2
2023-01-11T21:03:27.2025437Z skip: Need at least 2 CUDA devices (1.615s)
2023-01-11T21:03:27.2025784Z   test_summon_from_non_fsdp (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20517
2023-01-11T21:03:27.2026170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20518
2023-01-11T21:03:27.2026605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2026928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2027352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2027679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2028046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2028513Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2028836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2029244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2029580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2029912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2030386Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2030929Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2031313Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2031654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2031896Z dist init r=1, world=2
2023-01-11T21:03:27.2032074Z dist init r=0, world=2
2023-01-11T21:03:27.2032281Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:03:27.2032684Z   test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20545
2023-01-11T21:03:27.2033131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20546
2023-01-11T21:03:27.2033580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2033909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2034323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2034666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2034997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2035458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2035768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2036191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2036530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2036848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2037338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2037850Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2038232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2038558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2038807Z dist init r=0, world=2
2023-01-11T21:03:27.2038985Z dist init r=1, world=2
2023-01-11T21:03:27.2039177Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:03:27.2039590Z   test_summon_full_param_recursive_recurse_False_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20573
2023-01-11T21:03:27.2040035Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20574
2023-01-11T21:03:27.2040521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2040833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2041254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2041590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2041919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2042369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2042691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2043110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2043470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2043798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2044287Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2044802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2045170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2045515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2045767Z dist init r=0, world=2
2023-01-11T21:03:27.2045930Z dist init r=1, world=2
2023-01-11T21:03:27.2046135Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.2046547Z   test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20601
2023-01-11T21:03:27.2046990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20602
2023-01-11T21:03:27.2047425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2047747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2048168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2048504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2048821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2049284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2049609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2050016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2050350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2050680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2051372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2051877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2052264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2052607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2052929Z dist init r=1, world=2
2023-01-11T21:03:27.2053094Z dist init r=0, world=2
2023-01-11T21:03:27.2053302Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.2053719Z   test_summon_full_param_recursive_recurse_False_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20629
2023-01-11T21:03:27.2054149Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20630
2023-01-11T21:03:27.2054603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2054927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2055350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2055677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2056051Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2056522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2056830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2057249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2057586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2057915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2058455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2058974Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2059364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2059711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2059956Z dist init r=1, world=2
2023-01-11T21:03:27.2060139Z dist init r=0, world=2
2023-01-11T21:03:27.2060346Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.2060745Z   test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20657
2023-01-11T21:03:27.2061195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20658
2023-01-11T21:03:27.2061646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2061970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2062380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2062719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2063049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2063502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2063825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2064251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2064595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2064914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2065408Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2065972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2066359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2066692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2066949Z dist init r=1, world=2
2023-01-11T21:03:27.2067130Z dist init r=0, world=2
2023-01-11T21:03:27.2067325Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.2067734Z   test_summon_full_param_recursive_recurse_True_summon_outer_False_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20685
2023-01-11T21:03:27.2068180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20686
2023-01-11T21:03:27.2068679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2068996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2069421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2069766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2070089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2070556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2070883Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2071310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2071645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2071978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2072471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2072986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2073360Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2073707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2073967Z dist init r=0, world=2
2023-01-11T21:03:27.2074134Z dist init r=1, world=2
2023-01-11T21:03:27.2074340Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.2074756Z   test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20713
2023-01-11T21:03:27.2075201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20714
2023-01-11T21:03:27.2075633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2075959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2076380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2076707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2077037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2077499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2077856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2078269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2078606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2078934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2079409Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2079924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2080304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2080645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2080923Z dist init r=1, world=2
2023-01-11T21:03:27.2081104Z dist init r=0, world=2
2023-01-11T21:03:27.2081309Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.2081703Z   test_summon_full_param_recursive_recurse_True_summon_outer_True_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20741
2023-01-11T21:03:27.2082145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20742
2023-01-11T21:03:27.2082594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2082916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2083328Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2083665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2083997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2084461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2084775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2085198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2085536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2085853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2086340Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2086852Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2087238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2087567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2087818Z dist init r=0, world=2
2023-01-11T21:03:27.2087995Z dist init r=1, world=2
2023-01-11T21:03:27.2088187Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.2088567Z   test_summon_full_param_shard_value_mixed_precision_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20769
2023-01-11T21:03:27.2088986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20770
2023-01-11T21:03:27.2089437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2089747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2090172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2090550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2091035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2091532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2091859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2092286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2092612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2092948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2093495Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2094015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2094388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2094731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2094992Z dist init r=1, world=2
2023-01-11T21:03:27.2095158Z dist init r=0, world=2
2023-01-11T21:03:27.2095366Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.2095751Z   test_summon_full_param_shard_value_mixed_precision_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20797
2023-01-11T21:03:27.2096177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20798
2023-01-11T21:03:27.2096614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2096941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2097364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2097702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2098023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2098552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2098880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2099294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2099637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2099976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2100468Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2100973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2101359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2101711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2101966Z dist init r=1, world=2
2023-01-11T21:03:27.2102131Z dist init r=0, world=2
2023-01-11T21:03:27.2102337Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.2102692Z   test_summon_full_param_writeback (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20825
2023-01-11T21:03:27.2103132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20826
2023-01-11T21:03:27.2103584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2103911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2104324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2104663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2104995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2105461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2105774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2106234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2106577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2106907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2107382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2107900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2108280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2108613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2108865Z dist init r=0, world=2
2023-01-11T21:03:27.2109043Z dist init r=1, world=2
2023-01-11T21:03:27.2109248Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:03:27.2109639Z   test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20853
2023-01-11T21:03:27.2110070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20854
2023-01-11T21:03:27.2110517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2110831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2111255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2111595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2111927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2112378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2112710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2113131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2113471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2113788Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2114280Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2114796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2115164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2115505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2115792Z dist init r=1, world=2
2023-01-11T21:03:27.2115970Z dist init r=0, world=2
2023-01-11T21:03:27.2116168Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:03:27.2116564Z   test_summon_full_params_equivalence_rank0_only_False_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20881
2023-01-11T21:03:27.2116990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20882
2023-01-11T21:03:27.2117426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2117748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2118175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2118553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2118876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2119345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2119672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2120103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2120433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2120768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2121259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2121768Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2122160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2122507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2122764Z dist init r=0, world=2
2023-01-11T21:03:27.2122933Z dist init r=1, world=2
2023-01-11T21:03:27.2123141Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.2123538Z   test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_False (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20909
2023-01-11T21:03:27.2123956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20910
2023-01-11T21:03:27.2124408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2124736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2125165Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2125497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2125832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2126297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2126621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2127033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2127378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2127710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2128226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2128736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2129118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2129465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2129707Z dist init r=0, world=2
2023-01-11T21:03:27.2129888Z dist init r=1, world=2
2023-01-11T21:03:27.2130095Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:03:27.2130478Z   test_summon_full_params_equivalence_rank0_only_True_offload_to_cpu_True (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20937
2023-01-11T21:03:27.2131157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20938
2023-01-11T21:03:27.2131628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2131961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2132374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2132714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2133052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2133505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2133828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2134255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2134600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2134923Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2135417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2135936Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2136319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2136649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2136901Z dist init r=1, world=2
2023-01-11T21:03:27.2137080Z dist init r=0, world=2
2023-01-11T21:03:27.2137273Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:03:27.2137655Z   test_summon_full_params_respects_reshard_after_forward (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20965
2023-01-11T21:03:27.2138068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20966
2023-01-11T21:03:27.2138604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2138917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2139342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2139686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2140002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2140470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2140852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2141273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2141598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2141934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2142433Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2142950Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2143320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2143667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2143961Z dist init r=0, world=2
2023-01-11T21:03:27.2144130Z dist init r=1, world=2
2023-01-11T21:03:27.2144335Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:03:27.2144684Z   test_summon_single_param (__main__.TestSummonFullParams) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20993
2023-01-11T21:03:27.2145068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20994
2023-01-11T21:03:27.2145511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2145833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2146256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2146584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2146917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2147381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2147702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2148113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2148457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2148786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2149264Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2149775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2150159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2150503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2150746Z dist init r=1, world=2
2023-01-11T21:03:27.2150925Z dist init r=0, world=2
2023-01-11T21:03:27.2151129Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:03:27.2151366Z   test_with_grads_core (__main__.TestSummonFullParams)
2023-01-11T21:03:27.2151733Z Tests the core usage of ``summon_full_params(with_grads=True)``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21021
2023-01-11T21:03:27.2152116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21022
2023-01-11T21:03:27.2152566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2152877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2153303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2153680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2153996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2154461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2154782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2155203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2155528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2155856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2156388Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2156906Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2157275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2157615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2157872Z dist init r=1, world=2
2023-01-11T21:03:27.2158040Z dist init r=0, world=2
2023-01-11T21:03:27.2158244Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:03:27.2158504Z   test_with_grads_none_grads (__main__.TestSummonFullParams)
2023-01-11T21:03:27.2158996Z Tests that if all ranks' ``FlatParameter`` has ``None`` gradient, then ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21049
2023-01-11T21:03:27.2159377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21050
2023-01-11T21:03:27.2159828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2160151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2160561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2160901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2161232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2161697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2162008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2162429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2162769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2163089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:03:27.2163576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2164087Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:03:27.2164466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2164793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:03:27.2165052Z dist init r=1, world=2
2023-01-11T21:03:27.2165230Z dist init r=0, world=2
2023-01-11T21:03:27.2165421Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:03:27.2165798Z   test_summon_full_param_writeback (__main__.TestSummonFullParamsNoShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21077
2023-01-11T21:03:27.2166362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:03:27.2166685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:03:27.2167097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:03:27.2167434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:03:27.2167763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:03:27.2168251Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:03:27.2168620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:03:27.2168878Z dist init r=0, world=1
2023-01-11T21:03:27.2169120Z skip: Need at least 2 CUDA devices (1.508s)
2023-01-11T21:03:27.2169262Z 
2023-01-11T21:03:27.2169450Z ----------------------------------------------------------------------
2023-01-11T21:03:27.2169690Z Ran 52 tests in 84.067s
2023-01-11T21:03:27.2169803Z 
2023-01-11T21:03:27.2169874Z OK (skipped=52)
2023-01-11T21:03:27.2169981Z 
2023-01-11T21:03:27.2170062Z Generating XML reports...
2023-01-11T21:03:27.2170520Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20230111210202.xml
2023-01-11T21:03:27.2171445Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20230111210202.xml
2023-01-11T21:03:27.2171743Z 
2023-01-11T21:03:27.2172130Z ##[endgroup]
2023-01-11T21:03:27.2172750Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_summon_full_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_summon_full_params_t9w56nt6)
2023-01-11T21:03:27.2173036Z 
2023-01-11T21:03:27.2173228Z Running distributed/fsdp/test_fsdp_core ... [2023-01-11 21:03:27.179020]
2023-01-11T21:03:27.2173744Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:03:27.179426]
2023-01-11T21:05:04.1038628Z 
2023-01-11T21:05:04.1039448Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_core
2023-01-11T21:05:04.1040794Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_core (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_core_f30j_cdl)
2023-01-11T21:05:04.1041729Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_core
2023-01-11T21:05:04.1044890Z 
2023-01-11T21:05:04.1045126Z Running tests...
2023-01-11T21:05:04.1046890Z ----------------------------------------------------------------------
2023-01-11T21:05:04.1047496Z   test_pre_backward_hook_registration_after_state_dict (__main__.TestHooks)
2023-01-11T21:05:04.1048358Z Tests that FSDP pre-backward hooks are registered on forward pass ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21102
2023-01-11T21:05:04.1049042Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21103
2023-01-11T21:05:04.1049569Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21104
2023-01-11T21:05:04.1050116Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21105
2023-01-11T21:05:04.1051052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1051621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1052333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1052938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1053767Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1054572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1055146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1055894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1056443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1056963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1057745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1058242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1058904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1059334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1059666Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1060170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1060746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1061343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1061879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1062221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1062542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1063034Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1063547Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1064245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1064701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1065046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1065387Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1065727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1065977Z dist init r=2, world=4
2023-01-11T21:05:04.1066156Z dist init r=0, world=4
2023-01-11T21:05:04.1066333Z dist init r=1, world=4
2023-01-11T21:05:04.1066493Z dist init r=3, world=4
2023-01-11T21:05:04.1066693Z skip: Need at least 2 CUDA devices (2.070s)
2023-01-11T21:05:04.1066964Z   test_pre_backward_hook_registration_cuda_first_False (__main__.TestHooks)
2023-01-11T21:05:04.1067588Z Tests that FSDP pre-backward hooks are registered on forward pass ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21158
2023-01-11T21:05:04.1068145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21159
2023-01-11T21:05:04.1068493Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21160
2023-01-11T21:05:04.1068847Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21161
2023-01-11T21:05:04.1069357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1069771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1070265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1070597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1070990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1071459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1071848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1072259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1072665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1073040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1073578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1073907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1074370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1074710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1075082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1075552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1076666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1077098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1077427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1077758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1078249Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1078825Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1079326Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1079839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1080222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1080558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1080895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1081226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1081483Z dist init r=0, world=4
2023-01-11T21:05:04.1081648Z dist init r=2, world=4
2023-01-11T21:05:04.1081825Z dist init r=3, world=4
2023-01-11T21:05:04.1081998Z dist init r=1, world=4
2023-01-11T21:05:04.1082199Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1082521Z   test_pre_backward_hook_registration_cuda_first_True (__main__.TestHooks)
2023-01-11T21:05:04.1083024Z Tests that FSDP pre-backward hooks are registered on forward pass ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21214
2023-01-11T21:05:04.1083407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21215
2023-01-11T21:05:04.1083790Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21216
2023-01-11T21:05:04.1084109Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21217
2023-01-11T21:05:04.1084556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1084871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1085294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1085637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1085971Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1086420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1086777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1087202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1087527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1087858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1088319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1088639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1089046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1089450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1089781Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1090267Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1090784Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1091503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1091833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1092244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1092583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1092914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1093404Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1093908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1094290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1094635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1094975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1095301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1095552Z dist init r=0, world=4
2023-01-11T21:05:04.1095733Z dist init r=3, world=4
2023-01-11T21:05:04.1095896Z dist init r=1, world=4
2023-01-11T21:05:04.1096070Z dist init r=2, world=4
2023-01-11T21:05:04.1096276Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1096557Z   test_register_functions_called_cuda_first_False_mixed_precision_False (__main__.TestHooks)
2023-01-11T21:05:04.1097044Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21270
2023-01-11T21:05:04.1097434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21271
2023-01-11T21:05:04.1097762Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21272
2023-01-11T21:05:04.1098072Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21273
2023-01-11T21:05:04.1098522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1098924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1099339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1099730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1100068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1100535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1100846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1101265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1101606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1101924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1102387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1102786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1103215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1103543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1103879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1104341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1104668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1105076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1105418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1105750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1106233Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1106747Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1107261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1107771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1108141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1108487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1108822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1109163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1109446Z dist init r=1, world=4
2023-01-11T21:05:04.1109628Z dist init r=3, world=4
2023-01-11T21:05:04.1109805Z dist init r=0, world=4
2023-01-11T21:05:04.1109965Z dist init r=2, world=4
2023-01-11T21:05:04.1110167Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T21:05:04.1110454Z   test_register_functions_called_cuda_first_False_mixed_precision_True (__main__.TestHooks)
2023-01-11T21:05:04.1110844Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21326
2023-01-11T21:05:04.1111228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21327
2023-01-11T21:05:04.1111556Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21328
2023-01-11T21:05:04.1111876Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21329
2023-01-11T21:05:04.1112346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1112676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1113101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1113432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1113765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1114230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1114553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1114960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1115299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1115637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1116093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1116402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1116824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1117162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1117478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1117936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1118257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1118680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1119008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1119338Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1119833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1120467Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1120969Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1121480Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1121869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1122240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1122580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1122912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1123170Z dist init r=0, world=4
2023-01-11T21:05:04.1123340Z dist init r=2, world=4
2023-01-11T21:05:04.1123515Z dist init r=3, world=4
2023-01-11T21:05:04.1123690Z dist init r=1, world=4
2023-01-11T21:05:04.1123881Z skip: Need at least 2 CUDA devices (2.021s)
2023-01-11T21:05:04.1124167Z   test_register_functions_called_cuda_first_True_mixed_precision_False (__main__.TestHooks)
2023-01-11T21:05:04.1124567Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21382
2023-01-11T21:05:04.1124992Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21383
2023-01-11T21:05:04.1125306Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21384
2023-01-11T21:05:04.1125622Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21385
2023-01-11T21:05:04.1126070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1126384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1126809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1127157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1127488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1127942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1128270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1128691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1129101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1129432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1129897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1130218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1130626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1131130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1131468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1131924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1132245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1132664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1133005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1133323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1133813Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1134328Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1134934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1135429Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1135812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1136156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1136496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1136820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1137070Z dist init r=1, world=4
2023-01-11T21:05:04.1137249Z dist init r=0, world=4
2023-01-11T21:05:04.1137414Z dist init r=2, world=4
2023-01-11T21:05:04.1137591Z dist init r=3, world=4
2023-01-11T21:05:04.1137853Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T21:05:04.1138132Z   test_register_functions_called_cuda_first_True_mixed_precision_True (__main__.TestHooks)
2023-01-11T21:05:04.1138531Z Tests that ``_register_{pre|post}_backward_hooks()`` are called ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21438
2023-01-11T21:05:04.1138982Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21439
2023-01-11T21:05:04.1139305Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21440
2023-01-11T21:05:04.1139614Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21441
2023-01-11T21:05:04.1140060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1140389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1140801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1141141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1141477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1141939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1142249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1142669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1143009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1143328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1143792Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1144116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1144538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1144864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1145192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1145681Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1146171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1146483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1146904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1167304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1167732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1168271Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1168787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1169316Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1169704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1170043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1170390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1170974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1171266Z dist init r=1, world=4
2023-01-11T21:05:04.1171456Z dist init r=0, world=4
2023-01-11T21:05:04.1171636Z dist init r=2, world=4
2023-01-11T21:05:04.1171801Z dist init r=3, world=4
2023-01-11T21:05:04.1172012Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1172281Z   test_transformer_no_grad_mixed_precision_False (__main__.TestNoGrad)
2023-01-11T21:05:04.1172783Z Tests that for an FSDP-wrapped transformer model with shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21494
2023-01-11T21:05:04.1173168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21495
2023-01-11T21:05:04.1173502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21496
2023-01-11T21:05:04.1173831Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21497
2023-01-11T21:05:04.1174276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1174610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1175042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1175393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1175717Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1176181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1176494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1176908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1177232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1177554Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1178013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1178321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1178818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1179161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1179496Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1179980Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1180500Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1181169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1181544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1181980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1182327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1182663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1183142Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1183663Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1184109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1184459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1184792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1185135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1185392Z dist init r=0, world=4
2023-01-11T21:05:04.1185561Z dist init r=2, world=4
2023-01-11T21:05:04.1185740Z dist init r=1, world=4
2023-01-11T21:05:04.1185918Z dist init r=3, world=4
2023-01-11T21:05:04.1186110Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T21:05:04.1186376Z   test_transformer_no_grad_mixed_precision_True (__main__.TestNoGrad)
2023-01-11T21:05:04.1186868Z Tests that for an FSDP-wrapped transformer model with shared ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21550
2023-01-11T21:05:04.1187272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21551
2023-01-11T21:05:04.1187589Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21552
2023-01-11T21:05:04.1187911Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21553
2023-01-11T21:05:04.1188363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1188693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1189104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1189448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1189779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1190237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1190565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1190989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1191330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1191649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1192113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1192440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1192855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1193195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1193582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1194102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1194415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1194844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1195187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1195517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1195993Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1196511Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1197066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1197579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1197945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1198296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1198638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1198970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1199222Z dist init r=0, world=4
2023-01-11T21:05:04.1199406Z dist init r=3, world=4
2023-01-11T21:05:04.1199571Z dist init r=1, world=4
2023-01-11T21:05:04.1199749Z dist init r=2, world=4
2023-01-11T21:05:04.1199962Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1200226Z   test_param_change_after_init_mixed_precision_False (__main__.TestParamInit)
2023-01-11T21:05:04.1200741Z Tests that changing FSDP model parameter values in-place after FSDP ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21606
2023-01-11T21:05:04.1201146Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21607
2023-01-11T21:05:04.1201475Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21608
2023-01-11T21:05:04.1201787Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21609
2023-01-11T21:05:04.1202231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1202559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1202987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1203323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1203658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1204127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1204440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1204863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1205206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1205539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1205990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1206352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1206774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1207115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1207433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1207924Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1208416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1208725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1209149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1209528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1209865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1210347Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1211016Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1211553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1211935Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1212266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1212602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1212954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1213199Z dist init r=3, world=4
2023-01-11T21:05:04.1213380Z dist init r=1, world=4
2023-01-11T21:05:04.1213560Z dist init r=0, world=4
2023-01-11T21:05:04.1213725Z dist init r=2, world=4
2023-01-11T21:05:04.1213929Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1214204Z   test_param_change_after_init_mixed_precision_True (__main__.TestParamInit)
2023-01-11T21:05:04.1214718Z Tests that changing FSDP model parameter values in-place after FSDP ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21662
2023-01-11T21:05:04.1215109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21663
2023-01-11T21:05:04.1215435Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21664
2023-01-11T21:05:04.1215764Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21665
2023-01-11T21:05:04.1216204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1216534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1216959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1217302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1217622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1218088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1218414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1218922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1219317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1219653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1220150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1220659Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1221155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1221482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1221907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1222237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1222614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1223081Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1223411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1223826Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1224167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1224495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1224970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1225491Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1225880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1226227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1226557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1226895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1227153Z dist init r=0, world=4
2023-01-11T21:05:04.1227321Z dist init r=2, world=4
2023-01-11T21:05:04.1227499Z dist init r=1, world=4
2023-01-11T21:05:04.1227678Z dist init r=3, world=4
2023-01-11T21:05:04.1227871Z skip: Need at least 2 CUDA devices (1.721s)
2023-01-11T21:05:04.1228155Z   test_delayed_optim_step_offload_false_no_shard (__main__.TestParityWithDDP)
2023-01-11T21:05:04.1228557Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21718
2023-01-11T21:05:04.1228953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21719
2023-01-11T21:05:04.1229266Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21720
2023-01-11T21:05:04.1229591Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21721
2023-01-11T21:05:04.1230039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1230367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1230779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1231122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1231455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1231908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1232288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1232713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1233058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1233375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1233845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1234173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1234587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1234962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1235300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1235767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1236078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1236499Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1236843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1237178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1237656Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1238176Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1238689Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1239188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1239572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1239916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1240261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1240593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1240845Z dist init r=2, world=4
2023-01-11T21:05:04.1241025Z dist init r=1, world=4
2023-01-11T21:05:04.1241186Z dist init r=0, world=4
2023-01-11T21:05:04.1241365Z dist init r=3, world=4
2023-01-11T21:05:04.1241570Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T21:05:04.1241831Z   test_delayed_optim_step_offload_false_none (__main__.TestParityWithDDP)
2023-01-11T21:05:04.1242225Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21774
2023-01-11T21:05:04.1242618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21775
2023-01-11T21:05:04.1242947Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21776
2023-01-11T21:05:04.1243259Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21777
2023-01-11T21:05:04.1243707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1244037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1244466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1244831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1245164Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1245632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1245943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1246367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1246707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1247038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1247525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1247859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1248283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1248622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1248939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1249403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1249726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1250136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1250476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1250807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1252499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1253008Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1253518Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1254028Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1254393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1254735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1255073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1255419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1255659Z dist init r=2, world=4
2023-01-11T21:05:04.1255837Z dist init r=0, world=4
2023-01-11T21:05:04.1256012Z dist init r=3, world=4
2023-01-11T21:05:04.1256173Z dist init r=1, world=4
2023-01-11T21:05:04.1256373Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T21:05:04.1256658Z   test_delayed_optim_step_offload_false_shard_grad_op (__main__.TestParityWithDDP)
2023-01-11T21:05:04.1257048Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21830
2023-01-11T21:05:04.1257434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21831
2023-01-11T21:05:04.1257759Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21832
2023-01-11T21:05:04.1258086Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21833
2023-01-11T21:05:04.1258656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1258988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1259416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1259758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1260075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1260542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1260867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1261329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1261675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1262006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1262468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1262777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1263207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1263547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1263864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1264324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1264654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1265076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1265402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1265729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1266218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1266732Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1267228Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1267736Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1268119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1268460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1268787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1269121Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1269371Z dist init r=2, world=4
2023-01-11T21:05:04.1269537Z dist init r=1, world=4
2023-01-11T21:05:04.1269712Z dist init r=0, world=4
2023-01-11T21:05:04.1269887Z dist init r=3, world=4
2023-01-11T21:05:04.1270076Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1270352Z   test_delayed_optim_step_offload_true_no_shard (__main__.TestParityWithDDP)
2023-01-11T21:05:04.1271227Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82490 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2023-01-11T21:05:04.1271924Z   test_delayed_optim_step_offload_true_none (__main__.TestParityWithDDP)
2023-01-11T21:05:04.1272315Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21886
2023-01-11T21:05:04.1272693Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21887
2023-01-11T21:05:04.1273021Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21888
2023-01-11T21:05:04.1273348Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21889
2023-01-11T21:05:04.1273833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1274154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1274581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1274923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1275242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1275705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1276030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1276454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1276780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1277113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1277577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1277889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1278311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1278651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1278982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1279459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1279956Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1280286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1280711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1281037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1281371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1281863Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1282365Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1282869Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1283256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1283638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1283969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1284224Z dist init r=0, world=4
2023-01-11T21:05:04.1284409Z dist init r=3, world=4
2023-01-11T21:05:04.1284576Z dist init r=1, world=4
2023-01-11T21:05:04.1284840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1285100Z dist init r=2, world=4
2023-01-11T21:05:04.1285307Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T21:05:04.1285578Z   test_delayed_optim_step_offload_true_shard_grad_op (__main__.TestParityWithDDP)
2023-01-11T21:05:04.1285980Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21942
2023-01-11T21:05:04.1286372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21943
2023-01-11T21:05:04.1286723Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 21944
2023-01-11T21:05:04.1287048Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 21945
2023-01-11T21:05:04.1287496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1287830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1288246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1288591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1288922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1289371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1289702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1290127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1290467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1290784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1291440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1291769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1292198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1292529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1292865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1293330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1293642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1294065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1294489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1294819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1295298Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1295811Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1296326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1296905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1297273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1297618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1297959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1298288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1298544Z dist init r=1, world=4
2023-01-11T21:05:04.1298794Z dist init r=0, world=4
2023-01-11T21:05:04.1298971Z dist init r=3, world=4
2023-01-11T21:05:04.1299132Z dist init r=2, world=4
2023-01-11T21:05:04.1299338Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T21:05:04.1299674Z   test_delayed_reduce_scatter_offload_false_no_shard (__main__.TestParityWithDDP)
2023-01-11T21:05:04.1300066Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21998
2023-01-11T21:05:04.1300457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21999
2023-01-11T21:05:04.1300786Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22000
2023-01-11T21:05:04.1301098Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22001
2023-01-11T21:05:04.1301545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1301869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1302294Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1302625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1302958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1303420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1303741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1304149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1304490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1304821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1305271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1305591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1306017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1306355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1306669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1307159Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1307648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1307958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1308378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1308716Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1309083Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1309561Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1310075Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1310581Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1310957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1311285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1311621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1311991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1312238Z dist init r=1, world=4
2023-01-11T21:05:04.1312422Z dist init r=3, world=4
2023-01-11T21:05:04.1312600Z dist init r=2, world=4
2023-01-11T21:05:04.1312773Z dist init r=0, world=4
2023-01-11T21:05:04.1312970Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1313245Z   test_delayed_reduce_scatter_offload_false_none (__main__.TestParityWithDDP)
2023-01-11T21:05:04.1314102Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82704 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2023-01-11T21:05:04.1314706Z   test_delayed_reduce_scatter_offload_false_shard_grad_op (__main__.TestParityWithDDP)
2023-01-11T21:05:04.1315548Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82398 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2023-01-11T21:05:04.1316136Z   test_delayed_reduce_scatter_offload_true_no_shard (__main__.TestParityWithDDP)
2023-01-11T21:05:04.1316531Z Tests the FSDP forward, backward, and optimizer step runtime by ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22054
2023-01-11T21:05:04.1316925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22055
2023-01-11T21:05:04.1317237Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22056
2023-01-11T21:05:04.1317562Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22057
2023-01-11T21:05:04.1318011Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1318344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1318760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1319101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1319432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1319886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1320207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1320629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1320969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1321336Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1321799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1322119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1322528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1322862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1323193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1323654Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1323967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1324423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1324766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1325094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1325572Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1326093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1326605Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1327111Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1327478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1327826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1328164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1328407Z dist init r=1, world=4
2023-01-11T21:05:04.1328587Z dist init r=0, world=4
2023-01-11T21:05:04.1328761Z dist init r=3, world=4
2023-01-11T21:05:04.1329009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1329261Z dist init r=2, world=4
2023-01-11T21:05:04.1329465Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T21:05:04.1329739Z   test_delayed_reduce_scatter_offload_true_none (__main__.TestParityWithDDP)
2023-01-11T21:05:04.1330582Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82399 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2023-01-11T21:05:04.1331371Z   test_delayed_reduce_scatter_offload_true_shard_grad_op (__main__.TestParityWithDDP)
2023-01-11T21:05:04.1332232Z Tests the FSDP forward, backward, and optimizer step runtime by ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82403 for platform(s) linux, rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2023-01-11T21:05:04.1332907Z   test_mixture_of_experts_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22110
2023-01-11T21:05:04.1333314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22111
2023-01-11T21:05:04.1333685Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22112
2023-01-11T21:05:04.1334007Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22113
2023-01-11T21:05:04.1334460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1334778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1335207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1335552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1335888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1336347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1336716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1337148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1337497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1337817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1338288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1338678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1339095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1339438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1339771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1340240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1340552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1340977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1341317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1341639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1342135Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1342651Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1343168Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1343670Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1344052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1344398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1344740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1345075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1345331Z dist init r=0, world=4
2023-01-11T21:05:04.1345512Z dist init r=3, world=4
2023-01-11T21:05:04.1345679Z dist init r=1, world=4
2023-01-11T21:05:04.1345851Z dist init r=2, world=4
2023-01-11T21:05:04.1346059Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T21:05:04.1346414Z   test_mixture_of_experts_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22166
2023-01-11T21:05:04.1346854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22167
2023-01-11T21:05:04.1347182Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22168
2023-01-11T21:05:04.1347504Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22169
2023-01-11T21:05:04.1347940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1348265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1348687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1349029Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1349380Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1349848Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1350169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1350581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1350919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1351249Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1351711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1352024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1352447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1352792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1353111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1353574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1353901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1354321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1354644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1354977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1355465Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1355985Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1356486Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1356990Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1357373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1357713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1358041Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1358375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1358632Z dist init r=3, world=4
2023-01-11T21:05:04.1358831Z dist init r=2, world=4
2023-01-11T21:05:04.1359008Z dist init r=1, world=4
2023-01-11T21:05:04.1359182Z dist init r=0, world=4
2023-01-11T21:05:04.1359372Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1359746Z   test_mixture_of_experts_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22222
2023-01-11T21:05:04.1360151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22223
2023-01-11T21:05:04.1360479Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22224
2023-01-11T21:05:04.1360789Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22225
2023-01-11T21:05:04.1361231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1361557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1362021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1362368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1362697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1363159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1363469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1363888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1364228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1364541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1365000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1365324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1365742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1366063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1366394Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1366859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1367178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1367586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1367924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1368255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1368732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1369245Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1369757Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1370265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1370631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1371148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1371494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1371903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1372146Z dist init r=0, world=4
2023-01-11T21:05:04.1372327Z dist init r=2, world=4
2023-01-11T21:05:04.1372505Z dist init r=3, world=4
2023-01-11T21:05:04.1372668Z dist init r=1, world=4
2023-01-11T21:05:04.1372875Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1373244Z   test_mixture_of_experts_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22278
2023-01-11T21:05:04.1373638Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22279
2023-01-11T21:05:04.1373965Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22280
2023-01-11T21:05:04.1374286Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22281
2023-01-11T21:05:04.1374777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1375091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1375516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1375857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1376177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1376642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1376962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1377386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1377715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1378050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1378540Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1379114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1379425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1379852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1380193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1380510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1380978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1381304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1381726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1382051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1382383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1382873Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1383387Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1383882Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1384296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1384635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1384963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1385304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1385553Z dist init r=3, world=4
2023-01-11T21:05:04.1385730Z dist init r=2, world=4
2023-01-11T21:05:04.1385894Z dist init r=1, world=4
2023-01-11T21:05:04.1386069Z dist init r=0, world=4
2023-01-11T21:05:04.1386269Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T21:05:04.1386620Z   test_mixture_of_experts_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22334
2023-01-11T21:05:04.1387019Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22335
2023-01-11T21:05:04.1387373Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22336
2023-01-11T21:05:04.1387697Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22337
2023-01-11T21:05:04.1388132Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1388458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1388878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1389205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1389542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1390010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1390340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1390751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1391096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1391429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1391879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1392204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1392629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1392972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1393290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1393758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1394085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1394504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1394831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1395167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1395663Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1396165Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1396681Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1397221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1397608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1397940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1398280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1398622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1398880Z dist init r=2, world=4
2023-01-11T21:05:04.1399046Z dist init r=0, world=4
2023-01-11T21:05:04.1399222Z dist init r=1, world=4
2023-01-11T21:05:04.1399402Z dist init r=3, world=4
2023-01-11T21:05:04.1399594Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1400003Z   test_mixture_of_experts_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22390
2023-01-11T21:05:04.1400416Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22391
2023-01-11T21:05:04.1400732Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22392
2023-01-11T21:05:04.1401054Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22393
2023-01-11T21:05:04.1401500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1401825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1402236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1402580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1402915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1403368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1403690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1404112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1404455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1404774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1405232Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1405553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1405978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1406313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1406642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1407132Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1407610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1407933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1408357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1408697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1409017Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1409561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1410073Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1410583Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1411135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1411477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1411819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1412147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1412403Z dist init r=2, world=4
2023-01-11T21:05:04.1412633Z dist init r=0, world=4
2023-01-11T21:05:04.1412814Z dist init r=3, world=4
2023-01-11T21:05:04.1412979Z dist init r=1, world=4
2023-01-11T21:05:04.1413187Z skip: Need at least 2 CUDA devices (1.821s)
2023-01-11T21:05:04.1413575Z   test_mixture_of_experts_with_delay_before_free_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22446
2023-01-11T21:05:04.1413981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22447
2023-01-11T21:05:04.1414310Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22448
2023-01-11T21:05:04.1414636Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22449
2023-01-11T21:05:04.1415077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1415403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1415828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1416168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1416485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1416951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1417275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1417693Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1418019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1418349Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1418879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1419192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1419621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1419965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1420296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1420746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1421070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1421492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1421837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1422201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1422690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1423204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1423703Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1424206Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1424585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1424925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1425286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1425625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1425880Z dist init r=0, world=4
2023-01-11T21:05:04.1426047Z dist init r=1, world=4
2023-01-11T21:05:04.1426224Z dist init r=2, world=4
2023-01-11T21:05:04.1426396Z dist init r=3, world=4
2023-01-11T21:05:04.1426586Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T21:05:04.1426968Z   test_mixture_of_experts_with_delay_before_free_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22502
2023-01-11T21:05:04.1427382Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22503
2023-01-11T21:05:04.1427708Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22504
2023-01-11T21:05:04.1428023Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22505
2023-01-11T21:05:04.1428468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1428797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1429220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1429547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1429877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1430341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1430650Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1431071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1431418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1431748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1432196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1432520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1432940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1433281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1433595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1434083Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1434630Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1435107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1435429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1435846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1436184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1436502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1436988Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1437533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1437916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1438246Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1438578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1438922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1439164Z dist init r=0, world=4
2023-01-11T21:05:04.1439346Z dist init r=1, world=4
2023-01-11T21:05:04.1439521Z dist init r=2, world=4
2023-01-11T21:05:04.1439685Z dist init r=3, world=4
2023-01-11T21:05:04.1439889Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T21:05:04.1440283Z   test_mixture_of_experts_with_delay_before_free_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22558
2023-01-11T21:05:04.1440717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22559
2023-01-11T21:05:04.1441032Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22560
2023-01-11T21:05:04.1441357Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22561
2023-01-11T21:05:04.1441803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1442127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1442535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1442878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1443211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1443668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1443997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1444422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1444762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1445080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1445547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1445873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1446285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1446629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1446996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1447465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1447777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1448200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1448542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1448874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1449355Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1449877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1450429Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1451107Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1451493Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1451844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1452005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1452166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1452244Z dist init r=2, world=4
2023-01-11T21:05:04.1452304Z dist init r=3, world=4
2023-01-11T21:05:04.1452377Z dist init r=0, world=4
2023-01-11T21:05:04.1452450Z dist init r=1, world=4
2023-01-11T21:05:04.1452559Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T21:05:04.1452826Z   test_mixture_of_experts_with_delay_before_free_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22614
2023-01-11T21:05:04.1452981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22615
2023-01-11T21:05:04.1453134Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22616
2023-01-11T21:05:04.1453288Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22617
2023-01-11T21:05:04.1453553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1453678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1453957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1454093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1454270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1454541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1454664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1454943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1455065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1455239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1455505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1455627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1455959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1456092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1456267Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1456534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1456655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1456919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1457054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1457228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1457567Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1457869Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1458157Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1458452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1458679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1458843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1459015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1459241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1459321Z dist init r=3, world=4
2023-01-11T21:05:04.1459394Z dist init r=2, world=4
2023-01-11T21:05:04.1459467Z dist init r=0, world=4
2023-01-11T21:05:04.1459540Z dist init r=1, world=4
2023-01-11T21:05:04.1459644Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T21:05:04.1459892Z   test_mixture_of_experts_with_delay_before_free_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22670
2023-01-11T21:05:04.1460046Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22671
2023-01-11T21:05:04.1460200Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22672
2023-01-11T21:05:04.1460352Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22673
2023-01-11T21:05:04.1460629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1460754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1461032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1461168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1461343Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1461601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1461720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1461994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1462128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1462306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1462650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1462772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1463048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1463183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1463344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1463612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1463733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1464034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1464170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1464344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1464641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1464936Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1465224Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1465505Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1465671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1465837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1465997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1466155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1466230Z dist init r=3, world=4
2023-01-11T21:05:04.1466302Z dist init r=1, world=4
2023-01-11T21:05:04.1466373Z dist init r=2, world=4
2023-01-11T21:05:04.1466431Z dist init r=0, world=4
2023-01-11T21:05:04.1466533Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T21:05:04.1466796Z   test_mixture_of_experts_with_delay_before_free_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22726
2023-01-11T21:05:04.1466949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22727
2023-01-11T21:05:04.1467103Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22728
2023-01-11T21:05:04.1467256Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22729
2023-01-11T21:05:04.1467526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1467645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1467911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1468044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1468217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1468484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1468605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1468884Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1469046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1469221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1469475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1469595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1469873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1470005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1470177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1470472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1470596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1470874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1471006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1471168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1471463Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1471758Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1472049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1472345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1472508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1472669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1472829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1472988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1473049Z dist init r=3, world=4
2023-01-11T21:05:04.1473122Z dist init r=1, world=4
2023-01-11T21:05:04.1473193Z dist init r=2, world=4
2023-01-11T21:05:04.1473265Z dist init r=0, world=4
2023-01-11T21:05:04.1473366Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1473616Z   test_nested_always_wrap_model_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22782
2023-01-11T21:05:04.1473773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22783
2023-01-11T21:05:04.1473913Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22784
2023-01-11T21:05:04.1474065Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22785
2023-01-11T21:05:04.1474336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1474456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1474733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1474866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1475041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1475357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1475478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1475742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1475874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1476047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1476313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1476433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1476710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1476869Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1477044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1477310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1477417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1477692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1477826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1478000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1478297Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1478594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1478888Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1479180Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1479341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1479491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1479651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1479810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1479885Z dist init r=2, world=4
2023-01-11T21:05:04.1479959Z dist init r=0, world=4
2023-01-11T21:05:04.1480030Z dist init r=3, world=4
2023-01-11T21:05:04.1480105Z dist init r=1, world=4
2023-01-11T21:05:04.1480196Z skip: Need at least 2 CUDA devices (1.819s)
2023-01-11T21:05:04.1480438Z   test_nested_always_wrap_model_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22838
2023-01-11T21:05:04.1480592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22839
2023-01-11T21:05:04.1480743Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22840
2023-01-11T21:05:04.1480894Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22841
2023-01-11T21:05:04.1481163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1481282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1481564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1481725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1481888Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1482162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1482282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1482561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1482694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1482867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1483161Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1483457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1483566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1483839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1483972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1484144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1484408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1484527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1484801Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1484939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1485111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1485392Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1485684Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1485973Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1486136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1486296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1486457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1486535Z dist init r=0, world=4
2023-01-11T21:05:04.1486607Z dist init r=2, world=4
2023-01-11T21:05:04.1486666Z dist init r=3, world=4
2023-01-11T21:05:04.1486824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1486897Z dist init r=1, world=4
2023-01-11T21:05:04.1486998Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1487249Z   test_nested_always_wrap_model_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22894
2023-01-11T21:05:04.1487402Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22895
2023-01-11T21:05:04.1487552Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22896
2023-01-11T21:05:04.1487701Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22897
2023-01-11T21:05:04.1487960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1488109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1488384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1488517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1488690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1488954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1489074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1489349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1489480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1489674Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1489943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1490062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1490336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1490466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1490640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1491108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1491254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1491525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1491658Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1491832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1492131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1492426Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1492717Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1493006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1493171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1493336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1493498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1493646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1493720Z dist init r=0, world=4
2023-01-11T21:05:04.1493794Z dist init r=3, world=4
2023-01-11T21:05:04.1493866Z dist init r=1, world=4
2023-01-11T21:05:04.1493938Z dist init r=2, world=4
2023-01-11T21:05:04.1494042Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T21:05:04.1494285Z   test_nested_always_wrap_model_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22950
2023-01-11T21:05:04.1494429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22951
2023-01-11T21:05:04.1494582Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 22952
2023-01-11T21:05:04.1494790Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 22953
2023-01-11T21:05:04.1495064Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1495186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1495467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1495601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1495776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1496034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1496155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1496470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1496604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1496780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1497079Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1497347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1497468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1497743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1497862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1498038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1498304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1498427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1498766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1498901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1499075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1499374Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1499673Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1499953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1500116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1500281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1500442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1500603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1500678Z dist init r=0, world=4
2023-01-11T21:05:04.1500753Z dist init r=2, world=4
2023-01-11T21:05:04.1500830Z dist init r=3, world=4
2023-01-11T21:05:04.1500888Z dist init r=1, world=4
2023-01-11T21:05:04.1500993Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T21:05:04.1501239Z   test_nested_always_wrap_model_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23006
2023-01-11T21:05:04.1501429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23007
2023-01-11T21:05:04.1501586Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23008
2023-01-11T21:05:04.1501740Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23009
2023-01-11T21:05:04.1502015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1502140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1502405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1502540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1502742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1503015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1503135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1503413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1503547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1503722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1503989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1504098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1504376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1504515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1504689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1504958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1505080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1505356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1505487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1505646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1505945Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1506246Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1506534Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1506824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1506990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1507153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1507314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1507474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1507537Z dist init r=0, world=4
2023-01-11T21:05:04.1507608Z dist init r=1, world=4
2023-01-11T21:05:04.1507715Z dist init r=3, world=4
2023-01-11T21:05:04.1507787Z dist init r=2, world=4
2023-01-11T21:05:04.1507890Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1508139Z   test_nested_always_wrap_model_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23062
2023-01-11T21:05:04.1508292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23063
2023-01-11T21:05:04.1508443Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23064
2023-01-11T21:05:04.1508582Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23065
2023-01-11T21:05:04.1508855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1508976Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1509286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1509425Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1509596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1509864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1509984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1510249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1510379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1510551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1510819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1510940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1511213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1511347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1511517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1511785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1511893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1512168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1512299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1512474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1512769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1513061Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1513352Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1513639Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1513803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1513952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1514116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1514319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1514395Z dist init r=2, world=4
2023-01-11T21:05:04.1514467Z dist init r=0, world=4
2023-01-11T21:05:04.1514538Z dist init r=1, world=4
2023-01-11T21:05:04.1514610Z dist init r=3, world=4
2023-01-11T21:05:04.1514699Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T21:05:04.1514944Z   test_nested_wrapped_model_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23118
2023-01-11T21:05:04.1515098Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23119
2023-01-11T21:05:04.1515250Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23120
2023-01-11T21:05:04.1515401Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23121
2023-01-11T21:05:04.1515698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1515822Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1516103Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1516234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1516395Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1516662Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1516782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1517060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1517192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1517372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1517637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1517758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1518032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1518151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1518322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1518588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1518707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1518989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1519121Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1519293Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1519588Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1519869Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1520156Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1520445Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1520609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1520811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1520971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1521128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1521206Z dist init r=1, world=4
2023-01-11T21:05:04.1521279Z dist init r=2, world=4
2023-01-11T21:05:04.1521339Z dist init r=3, world=4
2023-01-11T21:05:04.1521411Z dist init r=0, world=4
2023-01-11T21:05:04.1521513Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1521750Z   test_nested_wrapped_model_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23174
2023-01-11T21:05:04.1521905Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23175
2023-01-11T21:05:04.1522085Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23176
2023-01-11T21:05:04.1522236Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23177
2023-01-11T21:05:04.1522510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1522620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1522900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1523036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1523213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1523483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1523607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1523890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1524023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1524185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1524454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1524575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1524852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1524984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1525157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1525428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1525550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1525830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1525950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1526122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1526417Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1526711Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1527005Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1527326Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1527489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1527648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1527809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1527953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1528028Z dist init r=3, world=4
2023-01-11T21:05:04.1528100Z dist init r=0, world=4
2023-01-11T21:05:04.1528172Z dist init r=1, world=4
2023-01-11T21:05:04.1528243Z dist init r=2, world=4
2023-01-11T21:05:04.1528345Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1528622Z   test_nested_wrapped_model_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23230
2023-01-11T21:05:04.1528768Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23231
2023-01-11T21:05:04.1528918Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23232
2023-01-11T21:05:04.1529070Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23233
2023-01-11T21:05:04.1529339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1529461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1529738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1529870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1530048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1530316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1530423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1530696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1530944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1531192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1531465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1531587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1531866Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1532002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1532163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1532430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1532553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1532832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1532965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1533143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1533439Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1533788Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1534080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1534359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1534523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1534685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1534845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1535004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1535080Z dist init r=0, world=4
2023-01-11T21:05:04.1535190Z dist init r=1, world=4
2023-01-11T21:05:04.1535264Z dist init r=3, world=4
2023-01-11T21:05:04.1535323Z dist init r=2, world=4
2023-01-11T21:05:04.1535427Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T21:05:04.1535673Z   test_nested_wrapped_model_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23286
2023-01-11T21:05:04.1535831Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23287
2023-01-11T21:05:04.1535985Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23288
2023-01-11T21:05:04.1536135Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23289
2023-01-11T21:05:04.1536408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1536529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1536794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1536927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1537099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1537365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1537485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1537763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1537896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1538069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1538365Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1538687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1538811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1539087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1539220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1539397Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1539664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1539787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1540063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1540228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1540387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1540687Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1540977Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1541266Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1541430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1541592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1541781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1541943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1542017Z dist init r=0, world=4
2023-01-11T21:05:04.1542077Z dist init r=3, world=4
2023-01-11T21:05:04.1542148Z dist init r=2, world=4
2023-01-11T21:05:04.1542219Z dist init r=1, world=4
2023-01-11T21:05:04.1542321Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1542557Z   test_nested_wrapped_model_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23342
2023-01-11T21:05:04.1542711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23343
2023-01-11T21:05:04.1542863Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23344
2023-01-11T21:05:04.1543001Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23345
2023-01-11T21:05:04.1543278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1543401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1543683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1543816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1543990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1544258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1544382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1544660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1544782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1544959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1545227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1545349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1545628Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1545759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1545931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1546228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1546484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1546635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1546906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1547039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1547210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1547504Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1547798Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1548084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1548275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1548426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1548584Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1548746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1548821Z dist init r=0, world=4
2023-01-11T21:05:04.1548893Z dist init r=1, world=4
2023-01-11T21:05:04.1548964Z dist init r=3, world=4
2023-01-11T21:05:04.1549034Z dist init r=2, world=4
2023-01-11T21:05:04.1549137Z skip: Need at least 2 CUDA devices (1.722s)
2023-01-11T21:05:04.1549375Z   test_nested_wrapped_model_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23398
2023-01-11T21:05:04.1549528Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23399
2023-01-11T21:05:04.1549686Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23400
2023-01-11T21:05:04.1549836Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23401
2023-01-11T21:05:04.1550108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1550229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1550508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1550641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1550802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1551069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1551192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1551472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1551604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1551776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1552072Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1552342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1552464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1552728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1552863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1553070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1553338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1553460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1553740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1553873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1554047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1554346Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1554664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1554961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1555126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1555290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1555451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1555612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1555691Z dist init r=0, world=4
2023-01-11T21:05:04.1555765Z dist init r=2, world=4
2023-01-11T21:05:04.1555824Z dist init r=1, world=4
2023-01-11T21:05:04.1555897Z dist init r=3, world=4
2023-01-11T21:05:04.1555999Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T21:05:04.1556280Z   test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23454
2023-01-11T21:05:04.1556441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23455
2023-01-11T21:05:04.1556594Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23456
2023-01-11T21:05:04.1556748Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23457
2023-01-11T21:05:04.1557019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1557128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1557411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1557545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1557725Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1557996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1558118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1558396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1558531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1558708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1558964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1559086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1559363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1559524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1559742Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1560089Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1560359Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1560481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1560746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1560877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1561119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1561416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1561709Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1562000Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1562163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1562329Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1562489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1562637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1562719Z dist init r=1, world=4
2023-01-11T21:05:04.1562795Z dist init r=0, world=4
2023-01-11T21:05:04.1562868Z dist init r=3, world=4
2023-01-11T21:05:04.1562941Z dist init r=2, world=4
2023-01-11T21:05:04.1563044Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1563315Z   test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23510
2023-01-11T21:05:04.1563470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23511
2023-01-11T21:05:04.1563610Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23512
2023-01-11T21:05:04.1563761Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23513
2023-01-11T21:05:04.1564030Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1564152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1564434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1564567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1564740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1565006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1565114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1565395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1565527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1565699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1566017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1566138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1566413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1566548Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1566716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1566971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1567093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1567366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1567526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1567703Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1567996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1568288Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1568577Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1568860Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1569012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1569177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1569337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1569495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1569569Z dist init r=3, world=4
2023-01-11T21:05:04.1569643Z dist init r=1, world=4
2023-01-11T21:05:04.1569714Z dist init r=2, world=4
2023-01-11T21:05:04.1569784Z dist init r=0, world=4
2023-01-11T21:05:04.1569874Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T21:05:04.1570156Z   test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23566
2023-01-11T21:05:04.1570308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23567
2023-01-11T21:05:04.1570462Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23568
2023-01-11T21:05:04.1570617Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23569
2023-01-11T21:05:04.1571042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1571188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1571471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1571593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1571764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1572033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1572154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1572429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1572614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1572789Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1573053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1573169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1573431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1573559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1573732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1573998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1574154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1574427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1574557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1574729Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1575012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1575306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1575592Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1575883Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1576047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1576204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1576362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1576517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1576589Z dist init r=0, world=4
2023-01-11T21:05:04.1576649Z dist init r=2, world=4
2023-01-11T21:05:04.1576723Z dist init r=3, world=4
2023-01-11T21:05:04.1576794Z dist init r=1, world=4
2023-01-11T21:05:04.1576896Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T21:05:04.1577169Z   test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23622
2023-01-11T21:05:04.1577323Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23623
2023-01-11T21:05:04.1577471Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23624
2023-01-11T21:05:04.1577617Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23625
2023-01-11T21:05:04.1577878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1577997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1578273Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1578407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1578644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1578921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1579072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1579351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1579472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1579644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1579913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1580035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1580308Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1580467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1580640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1580911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1581030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1581292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1581422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1581593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1581884Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1582185Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1582476Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1582764Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1582927Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1583085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1583232Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1583391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1583462Z dist init r=3, world=4
2023-01-11T21:05:04.1583531Z dist init r=1, world=4
2023-01-11T21:05:04.1583603Z dist init r=2, world=4
2023-01-11T21:05:04.1583675Z dist init r=0, world=4
2023-01-11T21:05:04.1583775Z skip: Need at least 2 CUDA devices (1.820s)
2023-01-11T21:05:04.1584048Z   test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23678
2023-01-11T21:05:04.1584189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23679
2023-01-11T21:05:04.1584339Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23680
2023-01-11T21:05:04.1584489Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23681
2023-01-11T21:05:04.1584757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1584879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1585156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1585320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1585495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1585750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1585872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1586145Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1586277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1586450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1586713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1586865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1587138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1587271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1587433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1587698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1587817Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1588093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1588225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1588400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1588698Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1588990Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1589279Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1589562Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1589723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1589880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1590040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1590202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1590275Z dist init r=0, world=4
2023-01-11T21:05:04.1590346Z dist init r=3, world=4
2023-01-11T21:05:04.1590416Z dist init r=1, world=4
2023-01-11T21:05:04.1590475Z dist init r=2, world=4
2023-01-11T21:05:04.1590577Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1590856Z   test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23734
2023-01-11T21:05:04.1591011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23735
2023-01-11T21:05:04.1591164Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23736
2023-01-11T21:05:04.1591318Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23737
2023-01-11T21:05:04.1591594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1591742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1592012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1592144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1592323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1592591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1592713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1592990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1593120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1593322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1593590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1593698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1593979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1594110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1594282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1594577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1594849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1594973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1595250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1595369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1595539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1595832Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1596127Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1596419Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1596581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1596743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1596902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1597064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1597128Z dist init r=0, world=4
2023-01-11T21:05:04.1597199Z dist init r=3, world=4
2023-01-11T21:05:04.1597270Z dist init r=1, world=4
2023-01-11T21:05:04.1597340Z dist init r=2, world=4
2023-01-11T21:05:04.1597442Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T21:05:04.1597674Z   test_transformer_offload_false_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23790
2023-01-11T21:05:04.1597829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23791
2023-01-11T21:05:04.1597980Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23792
2023-01-11T21:05:04.1598146Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23793
2023-01-11T21:05:04.1598420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1598540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1598817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1598951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1599125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1599392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1599512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1599804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1599938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1600111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1600378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1600497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1600774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1600906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1601080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1601351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1601459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1601735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1601868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1602038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1602334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1602629Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1602922Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1603216Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1603377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1603526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1603687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1603846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1603919Z dist init r=2, world=4
2023-01-11T21:05:04.1603989Z dist init r=0, world=4
2023-01-11T21:05:04.1604060Z dist init r=1, world=4
2023-01-11T21:05:04.1604130Z dist init r=3, world=4
2023-01-11T21:05:04.1604220Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T21:05:04.1604452Z   test_transformer_offload_false_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23846
2023-01-11T21:05:04.1604637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23847
2023-01-11T21:05:04.1604792Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23848
2023-01-11T21:05:04.1604942Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23849
2023-01-11T21:05:04.1605214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1605334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1605609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1605745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1605934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1606206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1606328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1606606Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1606744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1606918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1607188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1607308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1607569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1607706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1607879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1608147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1608269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1608545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1608678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1608851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1609149Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1609436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1609728Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1610022Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1610185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1610347Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1610509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1610669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1610744Z dist init r=1, world=4
2023-01-11T21:05:04.1610919Z dist init r=3, world=4
2023-01-11T21:05:04.1611098Z dist init r=0, world=4
2023-01-11T21:05:04.1611177Z dist init r=2, world=4
2023-01-11T21:05:04.1611281Z skip: Need at least 2 CUDA devices (1.819s)
2023-01-11T21:05:04.1611524Z   test_transformer_offload_false_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23902
2023-01-11T21:05:04.1611680Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23903
2023-01-11T21:05:04.1611833Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23904
2023-01-11T21:05:04.1611987Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23905
2023-01-11T21:05:04.1612252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1612377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1612700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1612839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1613014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1613284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1613406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1613683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1613818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1613981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1614252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1614376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1614653Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1614785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1614959Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1615228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1615354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1615635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1615755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1615932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1616233Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1616528Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1616819Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1617112Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1617275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1617439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1617601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1617782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1617858Z dist init r=0, world=4
2023-01-11T21:05:04.1617932Z dist init r=3, world=4
2023-01-11T21:05:04.1618003Z dist init r=1, world=4
2023-01-11T21:05:04.1618075Z dist init r=2, world=4
2023-01-11T21:05:04.1618178Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1618408Z   test_transformer_offload_true_no_shard (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23958
2023-01-11T21:05:04.1618549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23959
2023-01-11T21:05:04.1618778Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 23960
2023-01-11T21:05:04.1618930Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 23961
2023-01-11T21:05:04.1619238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1619364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1619643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1619778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1619954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1620226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1620333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1620610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1620744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1620924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1621221Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1621491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1621612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1621886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1622006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1622177Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1622443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1622567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1622840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1622973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1623149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1623442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1623736Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1624011Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1624175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1624366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1624526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1624683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1624757Z dist init r=3, world=4
2023-01-11T21:05:04.1624829Z dist init r=1, world=4
2023-01-11T21:05:04.1624900Z dist init r=2, world=4
2023-01-11T21:05:04.1624958Z dist init r=0, world=4
2023-01-11T21:05:04.1625061Z skip: Need at least 2 CUDA devices (1.719s)
2023-01-11T21:05:04.1625290Z   test_transformer_offload_true_none (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24014
2023-01-11T21:05:04.1625444Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24015
2023-01-11T21:05:04.1625622Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24016
2023-01-11T21:05:04.1625775Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24017
2023-01-11T21:05:04.1626053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1626174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1626439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1626574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1626748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1627014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1627133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1627415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1627547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1627722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1628016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1628272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1628390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1628665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1628795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1628974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1629242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1629361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1629636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1629757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1629930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1630225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1630517Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1630839Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1631003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1631163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1631322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1631480Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1631542Z dist init r=3, world=4
2023-01-11T21:05:04.1631614Z dist init r=1, world=4
2023-01-11T21:05:04.1631686Z dist init r=2, world=4
2023-01-11T21:05:04.1631758Z dist init r=0, world=4
2023-01-11T21:05:04.1631859Z skip: Need at least 2 CUDA devices (1.819s)
2023-01-11T21:05:04.1632124Z   test_transformer_offload_true_shard_grad_op (__main__.TestParityWithDDP) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24070
2023-01-11T21:05:04.1632281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24071
2023-01-11T21:05:04.1632432Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 24072
2023-01-11T21:05:04.1632570Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 24073
2023-01-11T21:05:04.1632843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1632965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1633244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1633377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1633549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:05:04.1633821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1633941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1634204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1634337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1634510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:05:04.1634803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1635071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1635191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1635467Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1635599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1635771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:05:04.1636024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:05:04.1636143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:05:04.1636419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:05:04.1636552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:05:04.1636723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:05:04.1637018Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1637342Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1637637Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:05:04.1637799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:05:04.1637947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:05:04.1638109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:05:04.1638269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:05:04.1638343Z dist init r=3, world=4
2023-01-11T21:05:04.1638415Z dist init r=1, world=4
2023-01-11T21:05:04.1638516Z dist init r=0, world=4
2023-01-11T21:05:04.1638588Z dist init r=2, world=4
2023-01-11T21:05:04.1638679Z skip: Need at least 2 CUDA devices (1.720s)
2023-01-11T21:05:04.1638700Z 
2023-01-11T21:05:04.1638888Z ----------------------------------------------------------------------
2023-01-11T21:05:04.1638966Z Ran 59 tests in 95.331s
2023-01-11T21:05:04.1638971Z 
2023-01-11T21:05:04.1639043Z OK (skipped=59)
2023-01-11T21:05:04.1639048Z 
2023-01-11T21:05:04.1639132Z Generating XML reports...
2023-01-11T21:05:04.1639439Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestHooks-20230111210328.xml
2023-01-11T21:05:04.1639743Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestNoGrad-20230111210328.xml
2023-01-11T21:05:04.1640049Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParamInit-20230111210328.xml
2023-01-11T21:05:04.1640370Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParityWithDDP-20230111210328.xml
2023-01-11T21:05:04.1640377Z 
2023-01-11T21:05:04.1640741Z ##[endgroup]
2023-01-11T21:05:04.1641080Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_core (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_core_f30j_cdl)
2023-01-11T21:05:04.1641085Z 
2023-01-11T21:05:04.1641298Z Running distributed/fsdp/test_fsdp_mixed_precision ... [2023-01-11 21:05:04.106277]
2023-01-11T21:05:04.1641676Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:05:04.106632]
2023-01-11T21:06:43.7321116Z 
2023-01-11T21:06:43.7321928Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_mixed_precision
2023-01-11T21:06:43.7322979Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_a41s99uk)
2023-01-11T21:06:43.7324007Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision
2023-01-11T21:06:43.7324359Z 
2023-01-11T21:06:43.7324485Z Running tests...
2023-01-11T21:06:43.7333019Z ----------------------------------------------------------------------
2023-01-11T21:06:43.7333815Z   test_float16_on_one_submodule (__main__.TestFSDPDifferentSubmodulePrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24137
2023-01-11T21:06:43.7334723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24138
2023-01-11T21:06:43.7335510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7336079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7336816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7337649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7338199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7339089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7339642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7340338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7340927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7341553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7342244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7342941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7343340Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7343689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7343952Z dist init r=0, world=2
2023-01-11T21:06:43.7344120Z dist init r=1, world=2
2023-01-11T21:06:43.7344329Z skip: Need at least 2 CUDA devices (1.925s)
2023-01-11T21:06:43.7344733Z   test_float16_on_one_submodule_skip_inputs (__main__.TestFSDPDifferentSubmodulePrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24165
2023-01-11T21:06:43.7345155Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24166
2023-01-11T21:06:43.7345613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7345946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7346375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7346707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7347040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7347505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7347821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7348246Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7348587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7348919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7349403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7349921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7350302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7350649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7350895Z dist init r=0, world=2
2023-01-11T21:06:43.7351083Z dist init r=1, world=2
2023-01-11T21:06:43.7351292Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7351688Z   test_float16_on_one_submodule_skip_inputs_error (__main__.TestFSDPDifferentSubmodulePrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24193
2023-01-11T21:06:43.7352130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24194
2023-01-11T21:06:43.7352634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7352962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7353376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7353720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7354055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7354509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7354833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7355256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7355634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7355958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7356456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7356971Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7357358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7357693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7357946Z dist init r=1, world=2
2023-01-11T21:06:43.7358128Z dist init r=0, world=2
2023-01-11T21:06:43.7358324Z skip: Need at least 2 CUDA devices (1.914s)
2023-01-11T21:06:43.7358731Z   test_submodules_with_different_precisions (__main__.TestFSDPDifferentSubmodulePrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24221
2023-01-11T21:06:43.7359170Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24222
2023-01-11T21:06:43.7359608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7359938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7360365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7360710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7361033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7361494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7361824Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7362242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7362581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7362908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7363394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7363899Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7364278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7364619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7364905Z dist init r=1, world=2
2023-01-11T21:06:43.7365082Z dist init r=0, world=2
2023-01-11T21:06:43.7365285Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7365694Z   test_submodules_with_different_precisions_error (__main__.TestFSDPDifferentSubmodulePrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24249
2023-01-11T21:06:43.7366125Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24250
2023-01-11T21:06:43.7366571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7366894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7367307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7367647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7368019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7368488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7368796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7369216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7369556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7369883Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7370354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7371033Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7371422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7371754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7372014Z dist init r=1, world=2
2023-01-11T21:06:43.7372196Z dist init r=0, world=2
2023-01-11T21:06:43.7372405Z skip: Need at least 2 CUDA devices (1.914s)
2023-01-11T21:06:43.7372793Z   test_submodules_with_external_inputs (__main__.TestFSDPDifferentSubmodulePrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24277
2023-01-11T21:06:43.7373227Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24278
2023-01-11T21:06:43.7373679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7373999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7374428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7374772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7375102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7375547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7375869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7376292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7376627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7376942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7377432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7378014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7378382Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7378729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7379057Z dist init r=0, world=2
2023-01-11T21:06:43.7379238Z dist init r=1, world=2
2023-01-11T21:06:43.7379432Z skip: Need at least 2 CUDA devices (1.914s)
2023-01-11T21:06:43.7379843Z   test_mixed_precision_with_ignored_module (__main__.TestFSDPMixedPrecisionIgnoredModules) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24305
2023-01-11T21:06:43.7380399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7380768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7381196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7381534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7381862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7382338Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:06:43.7382718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7382972Z dist init r=0, world=1
2023-01-11T21:06:43.7383163Z skip: Need at least 1 CUDA device (1.809s)
2023-01-11T21:06:43.7383543Z   test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24319
2023-01-11T21:06:43.7383966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24320
2023-01-11T21:06:43.7384416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7384726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7385152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7385495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7385827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7386278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7386598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7387018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7387346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7387675Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7388162Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7388680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7389052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7389397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7389655Z dist init r=0, world=2
2023-01-11T21:06:43.7389823Z dist init r=1, world=2
2023-01-11T21:06:43.7390034Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7390378Z   test_input_grads_with_param_mixed_precision (__main__.TestFSDPMixedPrecisionSharded)
2023-01-11T21:06:43.7390807Z Tests that input tensors that require gradients do get their gradients ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24347
2023-01-11T21:06:43.7391194Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24348
2023-01-11T21:06:43.7391648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7391975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7392386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7392729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7393065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7393570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7393882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7394304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7394647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7394979Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7395457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7395974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7396358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7396697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7396958Z dist init r=1, world=2
2023-01-11T21:06:43.7397140Z dist init r=0, world=2
2023-01-11T21:06:43.7397354Z skip: Need at least 2 CUDA devices (2.014s)
2023-01-11T21:06:43.7397781Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24375
2023-01-11T21:06:43.7398248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24376
2023-01-11T21:06:43.7398695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7399008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7399427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7399775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7400104Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7400550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7400870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7401289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7401625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7401939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7402429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7402981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7403348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7403692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7403940Z dist init r=0, world=2
2023-01-11T21:06:43.7404121Z dist init r=1, world=2
2023-01-11T21:06:43.7404316Z skip: Need at least 2 CUDA devices (2.014s)
2023-01-11T21:06:43.7404728Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24403
2023-01-11T21:06:43.7405168Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24404
2023-01-11T21:06:43.7405604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7405972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7406398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7406735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7407052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7407514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7407835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7408253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7408575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7408907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7409400Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7409897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7410276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7410619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7410999Z dist init r=0, world=2
2023-01-11T21:06:43.7411166Z dist init r=1, world=2
2023-01-11T21:06:43.7411373Z skip: Need at least 2 CUDA devices (2.014s)
2023-01-11T21:06:43.7411811Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24431
2023-01-11T21:06:43.7412269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24432
2023-01-11T21:06:43.7412719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7413045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7413470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7413800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7414130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7414590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7414912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7415322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7415720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7416048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7416527Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7417046Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7417428Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7417774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7418023Z dist init r=0, world=2
2023-01-11T21:06:43.7418204Z dist init r=1, world=2
2023-01-11T21:06:43.7418471Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7418936Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24459
2023-01-11T21:06:43.7419386Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24460
2023-01-11T21:06:43.7419838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7420163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7420880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7421221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7421555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7422019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7422336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7422758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7423097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7423414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7423901Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7424410Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7424792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7425128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7425374Z dist init r=1, world=2
2023-01-11T21:06:43.7425552Z dist init r=0, world=2
2023-01-11T21:06:43.7425744Z skip: Need at least 2 CUDA devices (1.914s)
2023-01-11T21:06:43.7426183Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24487
2023-01-11T21:06:43.7426647Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24488
2023-01-11T21:06:43.7427093Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7427402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7427821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7428212Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7428546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7429000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7429328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7429752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7430081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7430416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7430906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7431463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7431839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7432178Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7432433Z dist init r=1, world=2
2023-01-11T21:06:43.7432610Z dist init r=0, world=2
2023-01-11T21:06:43.7432806Z skip: Need at least 2 CUDA devices (1.918s)
2023-01-11T21:06:43.7433217Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24515
2023-01-11T21:06:43.7433664Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24516
2023-01-11T21:06:43.7434102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7434429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7434850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7435191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7435509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7435972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7436294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7436703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7437039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7437369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7437859Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7438362Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7438744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7439082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7439339Z dist init r=0, world=2
2023-01-11T21:06:43.7439507Z dist init r=1, world=2
2023-01-11T21:06:43.7439710Z skip: Need at least 2 CUDA devices (2.014s)
2023-01-11T21:06:43.7440145Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24543
2023-01-11T21:06:43.7440637Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24544
2023-01-11T21:06:43.7441087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7441411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7441836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7442166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7442497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7442962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7443271Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7443732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7444078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7444409Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7444887Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7445411Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7445797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7446144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7446395Z dist init r=1, world=2
2023-01-11T21:06:43.7446577Z dist init r=0, world=2
2023-01-11T21:06:43.7446786Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7447191Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24571
2023-01-11T21:06:43.7447639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24572
2023-01-11T21:06:43.7448101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7448429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7448842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7449185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7449519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7449973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7450296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7450721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7451198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7451518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7452016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7452531Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7452917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7453251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7453559Z dist init r=0, world=2
2023-01-11T21:06:43.7453742Z dist init r=1, world=2
2023-01-11T21:06:43.7453937Z skip: Need at least 2 CUDA devices (1.914s)
2023-01-11T21:06:43.7454374Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24599
2023-01-11T21:06:43.7454848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24600
2023-01-11T21:06:43.7455303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7455620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7456048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7456435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7456763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7457228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7457549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7457969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7458293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7458624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7459311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7459832Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7460206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7460553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7460810Z dist init r=1, world=2
2023-01-11T21:06:43.7460976Z dist init r=0, world=2
2023-01-11T21:06:43.7461183Z skip: Need at least 2 CUDA devices (1.914s)
2023-01-11T21:06:43.7461600Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24627
2023-01-11T21:06:43.7462047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24628
2023-01-11T21:06:43.7462483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7462807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7463230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7463557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7463890Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7464349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7464670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7465077Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7465412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7465741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7466278Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7466779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7467162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7467507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7467754Z dist init r=0, world=2
2023-01-11T21:06:43.7467931Z dist init r=1, world=2
2023-01-11T21:06:43.7468139Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7468562Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24655
2023-01-11T21:06:43.7469068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24656
2023-01-11T21:06:43.7469517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7469841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7470249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7470588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7470916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7471378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7471691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7472114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7472455Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7472783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7473257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7473771Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7474150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7474476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7474730Z dist init r=1, world=2
2023-01-11T21:06:43.7474910Z dist init r=0, world=2
2023-01-11T21:06:43.7475102Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7475518Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24683
2023-01-11T21:06:43.7475961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24684
2023-01-11T21:06:43.7476406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7476719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7477140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7477480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7477807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7478254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7478613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7479032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7479369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7479682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7480172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7480686Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7481053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7481432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7481685Z dist init r=0, world=2
2023-01-11T21:06:43.7481864Z dist init r=1, world=2
2023-01-11T21:06:43.7482057Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7482493Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24711
2023-01-11T21:06:43.7482955Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24712
2023-01-11T21:06:43.7483395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7483720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7484148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7484495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7484817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7485278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7485605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7486017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7486356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7486685Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7487171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7487676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7488063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7488406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7488662Z dist init r=1, world=2
2023-01-11T21:06:43.7488828Z dist init r=0, world=2
2023-01-11T21:06:43.7489030Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7489441Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24739
2023-01-11T21:06:43.7489870Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24740
2023-01-11T21:06:43.7490318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7490644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7491220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7491550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7491885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7492349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7492661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7493087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7493431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7493762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7494290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7494810Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7495193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7495538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7495787Z dist init r=1, world=2
2023-01-11T21:06:43.7495967Z dist init r=0, world=2
2023-01-11T21:06:43.7496178Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7496601Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24767
2023-01-11T21:06:43.7497072Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24768
2023-01-11T21:06:43.7497527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7497854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7498265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7498608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7498998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7499460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7499788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7500220Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7500565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7500889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7501380Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7501902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7502286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7502618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7502873Z dist init r=0, world=2
2023-01-11T21:06:43.7503055Z dist init r=1, world=2
2023-01-11T21:06:43.7503250Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7503714Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24795
2023-01-11T21:06:43.7504163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24796
2023-01-11T21:06:43.7504616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7504931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7505360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7505704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7506025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7506538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7506864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7507291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7507619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7507952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7508448Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7508961Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7509330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7509678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7509938Z dist init r=0, world=2
2023-01-11T21:06:43.7510108Z dist init r=1, world=2
2023-01-11T21:06:43.7510318Z skip: Need at least 2 CUDA devices (1.914s)
2023-01-11T21:06:43.7510765Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24823
2023-01-11T21:06:43.7511732Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24824
2023-01-11T21:06:43.7512188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7512500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7512923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7513267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7513593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7514062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7514391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7514815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7515141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7515466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7515952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7516466Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7516874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7517214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7517465Z dist init r=0, world=2
2023-01-11T21:06:43.7517630Z dist init r=1, world=2
2023-01-11T21:06:43.7517833Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7518259Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24851
2023-01-11T21:06:43.7518711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24852
2023-01-11T21:06:43.7519149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7519504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7519930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7520258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7520591Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7521049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7521371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7521780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7522117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7522447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7522940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7523441Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7523821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7524162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7524404Z dist init r=1, world=2
2023-01-11T21:06:43.7524582Z dist init r=0, world=2
2023-01-11T21:06:43.7524787Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7525240Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24879
2023-01-11T21:06:43.7525774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24880
2023-01-11T21:06:43.7526226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7526552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7526977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7527305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7527638Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7528098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7528407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7528829Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7529219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7529548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7530022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7530532Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7531029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7531375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7531621Z dist init r=0, world=2
2023-01-11T21:06:43.7531801Z dist init r=1, world=2
2023-01-11T21:06:43.7532009Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7532479Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24907
2023-01-11T21:06:43.7532938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24908
2023-01-11T21:06:43.7533391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7533720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7534129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7534472Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7534811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7535266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7535590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7536014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7536355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7536672Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7537161Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7537675Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7538057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7538389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7538646Z dist init r=0, world=2
2023-01-11T21:06:43.7538822Z dist init r=1, world=2
2023-01-11T21:06:43.7539088Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7539539Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24935
2023-01-11T21:06:43.7540024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24936
2023-01-11T21:06:43.7540484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7540799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7541228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7541625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7541942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7542406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7542733Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7543158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7543490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7543823Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7544316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7544853Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7545238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7545585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7545838Z dist init r=1, world=2
2023-01-11T21:06:43.7546003Z dist init r=0, world=2
2023-01-11T21:06:43.7546206Z skip: Need at least 2 CUDA devices (1.914s)
2023-01-11T21:06:43.7546630Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24963
2023-01-11T21:06:43.7547082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24964
2023-01-11T21:06:43.7547522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7547848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7548270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7548598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7548927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7549393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7549713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7550124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7550467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7550798Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7551275Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7551786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7552172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7552512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7552762Z dist init r=0, world=2
2023-01-11T21:06:43.7552939Z dist init r=1, world=2
2023-01-11T21:06:43.7553143Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7553581Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24991
2023-01-11T21:06:43.7554097Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24992
2023-01-11T21:06:43.7554552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7554882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7555293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7555642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7555974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7556440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7556753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7557211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7557561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7557880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7558374Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7558897Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7559284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7559619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7559874Z dist init r=0, world=2
2023-01-11T21:06:43.7560056Z dist init r=1, world=2
2023-01-11T21:06:43.7560253Z skip: Need at least 2 CUDA devices (1.914s)
2023-01-11T21:06:43.7560689Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25019
2023-01-11T21:06:43.7561152Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25020
2023-01-11T21:06:43.7561609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7561926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7562353Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7562697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7563032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7563485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7563811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7564236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7564564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7564891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7565383Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7565903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7566268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7566649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7566903Z dist init r=1, world=2
2023-01-11T21:06:43.7567081Z dist init r=0, world=2
2023-01-11T21:06:43.7567277Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7567719Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25047
2023-01-11T21:06:43.7568188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25048
2023-01-11T21:06:43.7568626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7568950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7569372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7569745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7570069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7570531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7570955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7571370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7571711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7572048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7572537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7573046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7573429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7573776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7574035Z dist init r=1, world=2
2023-01-11T21:06:43.7574202Z dist init r=0, world=2
2023-01-11T21:06:43.7574407Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7574825Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25075
2023-01-11T21:06:43.7575267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25076
2023-01-11T21:06:43.7575714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7576047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7576470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7576799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7577132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7577594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7577902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7578323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7578662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7579059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7579612Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7580126Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7580514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7580859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7581107Z dist init r=0, world=2
2023-01-11T21:06:43.7581287Z dist init r=1, world=2
2023-01-11T21:06:43.7581493Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7581962Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25103
2023-01-11T21:06:43.7582443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25104
2023-01-11T21:06:43.7582894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7583220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7583630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7583970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7584299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7584779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7585275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7585602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7586033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7586363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7586693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7587183Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7587568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7587901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7588151Z dist init r=0, world=2
2023-01-11T21:06:43.7588330Z dist init r=1, world=2
2023-01-11T21:06:43.7588527Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7588949Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25131
2023-01-11T21:06:43.7589398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25132
2023-01-11T21:06:43.7589845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7590155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7590576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7590915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7591232Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7591750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7592073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7592492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7592819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7593146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7593635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7594153Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7594520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7594904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7595167Z dist init r=1, world=2
2023-01-11T21:06:43.7595331Z dist init r=0, world=2
2023-01-11T21:06:43.7595535Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7595974Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25159
2023-01-11T21:06:43.7596446Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25160
2023-01-11T21:06:43.7596883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7597207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7597629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7597959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7598288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7598748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7599072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7599480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7599817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7600148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7600635Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7601141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7601524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7601867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7602111Z dist init r=1, world=2
2023-01-11T21:06:43.7602289Z dist init r=0, world=2
2023-01-11T21:06:43.7602492Z skip: Need at least 2 CUDA devices (2.013s)
2023-01-11T21:06:43.7602908Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25187
2023-01-11T21:06:43.7603345Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25188
2023-01-11T21:06:43.7603791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7604154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7604569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7604907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7605236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7605700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7606010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7606428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7606767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7607134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7607610Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7608124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7608505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7608833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7609091Z dist init r=0, world=2
2023-01-11T21:06:43.7609275Z dist init r=1, world=2
2023-01-11T21:06:43.7609479Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7609917Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25215
2023-01-11T21:06:43.7610401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25216
2023-01-11T21:06:43.7610975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7611293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7611718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7612067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7612405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7612857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7613188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7613618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7613962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7614280Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7614775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7615296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7615671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7616018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7616274Z dist init r=0, world=2
2023-01-11T21:06:43.7616458Z dist init r=1, world=2
2023-01-11T21:06:43.7616710Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7617130Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25243
2023-01-11T21:06:43.7617585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25244
2023-01-11T21:06:43.7618026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7618353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7618779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7619185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7619508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7620035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7620363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7620788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7621118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7621453Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7621949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7622455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7622838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7623192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7623451Z dist init r=0, world=2
2023-01-11T21:06:43.7623625Z dist init r=1, world=2
2023-01-11T21:06:43.7623830Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7624225Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25271
2023-01-11T21:06:43.7624640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25272
2023-01-11T21:06:43.7625087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7625416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7625845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7626176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7626513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7626976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7627299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7627707Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7628047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7628378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7628854Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7629375Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7629790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7630135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7630378Z dist init r=1, world=2
2023-01-11T21:06:43.7630554Z dist init r=0, world=2
2023-01-11T21:06:43.7630758Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7631025Z   test_mixed_precision_resnet (__main__.TestFSDPMixedPrecisionSharded)
2023-01-11T21:06:43.7631356Z End to end test to ensure mixed precision + auto_wrap works ... skip: no torchvision (0.001s)
2023-01-11T21:06:43.7631788Z   test_mp_batchnorm_convert_sync_bn_False (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25299
2023-01-11T21:06:43.7632205Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25300
2023-01-11T21:06:43.7632675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7633002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7633432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7633762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7634093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7634557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7634879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7635292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7635636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7635965Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7636453Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7636958Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7637338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7637683Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7637925Z dist init r=0, world=2
2023-01-11T21:06:43.7638102Z dist init r=1, world=2
2023-01-11T21:06:43.7638307Z skip: Need at least 2 CUDA devices (1.914s)
2023-01-11T21:06:43.7638701Z   test_mp_batchnorm_convert_sync_bn_True (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25327
2023-01-11T21:06:43.7639110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25328
2023-01-11T21:06:43.7639557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7639881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7640292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7640633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7640966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7641426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7641737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7642197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7642538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7642854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7643339Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7643852Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7644234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7644565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7644821Z dist init r=1, world=2
2023-01-11T21:06:43.7645032Z dist init r=0, world=2
2023-01-11T21:06:43.7645228Z skip: Need at least 2 CUDA devices (1.915s)
2023-01-11T21:06:43.7645601Z   test_mp_embedding_default (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25355
2023-01-11T21:06:43.7646012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25356
2023-01-11T21:06:43.7646461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7646779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7647200Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7647539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7647868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7648326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7648646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7649068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7649394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7649722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7650213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7650730Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7651218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7651567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7651825Z dist init r=0, world=2
2023-01-11T21:06:43.7651992Z dist init r=1, world=2
2023-01-11T21:06:43.7652205Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7652598Z   test_mp_embedding_only_params_and_bufs (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25383
2023-01-11T21:06:43.7653028Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25384
2023-01-11T21:06:43.7653478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7653813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7654239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7654651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7654975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7655444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7655767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7656181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7656522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7656860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7657356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7657907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7658295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7658638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7658974Z dist init r=1, world=2
2023-01-11T21:06:43.7659161Z dist init r=0, world=2
2023-01-11T21:06:43.7659368Z skip: Need at least 2 CUDA devices (2.013s)
2023-01-11T21:06:43.7659761Z   test_mp_embedding_params_and_reduce_diff (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25411
2023-01-11T21:06:43.7660177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25412
2023-01-11T21:06:43.7660635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7660969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7661382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7661724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7662059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7662529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7662838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7663259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7663599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7663934Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7664414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7664932Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7665322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7665654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7665914Z dist init r=1, world=2
2023-01-11T21:06:43.7666096Z dist init r=0, world=2
2023-01-11T21:06:43.7666308Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7666672Z   test_mp_embedding_reduce (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25439
2023-01-11T21:06:43.7667089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25440
2023-01-11T21:06:43.7667584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7667898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7668322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7668665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7669002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7669464Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7669791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7670226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7670604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7670931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:06:43.7671432Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7671951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:06:43.7672322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7672670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:06:43.7672922Z dist init r=0, world=2
2023-01-11T21:06:43.7673105Z dist init r=1, world=2
2023-01-11T21:06:43.7673300Z skip: Need at least 2 CUDA devices (1.913s)
2023-01-11T21:06:43.7673686Z   test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25467
2023-01-11T21:06:43.7674229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7674546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7674971Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7675313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7675648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7676122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:06:43.7676498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7676757Z dist init r=0, world=1
2023-01-11T21:06:43.7676951Z skip: Need at least 1 CUDA device (1.808s)
2023-01-11T21:06:43.7677334Z   test_mixed_precision_e2e_full_shard (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25481
2023-01-11T21:06:43.7677876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7678202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7678614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7678955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7679288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7679778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:06:43.7680202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7680459Z dist init r=0, world=1
2023-01-11T21:06:43.7680667Z skip: Need at least 1 CUDA device (1.809s)
2023-01-11T21:06:43.7681052Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25495
2023-01-11T21:06:43.7681604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:06:43.7681929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:06:43.7682354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:06:43.7682681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:06:43.7683081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:06:43.7683579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:06:43.7683950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:06:43.7684209Z dist init r=0, world=1
2023-01-11T21:06:43.7684415Z skip: Need at least 1 CUDA device (1.809s)
2023-01-11T21:06:43.7684553Z 
2023-01-11T21:06:43.7684753Z ----------------------------------------------------------------------
2023-01-11T21:06:43.7684983Z Ran 52 tests in 97.789s
2023-01-11T21:06:43.7685098Z 
2023-01-11T21:06:43.7685172Z OK (skipped=52)
2023-01-11T21:06:43.7685280Z 
2023-01-11T21:06:43.7685364Z Generating XML reports...
2023-01-11T21:06:43.7685884Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPDifferentSubmodulePrecision-20230111210505.xml
2023-01-11T21:06:43.7686609Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionIgnoredModules-20230111210505.xml
2023-01-11T21:06:43.7687298Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20230111210505.xml
2023-01-11T21:06:43.7687967Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20230111210505.xml
2023-01-11T21:06:43.7688259Z 
2023-01-11T21:06:43.7688623Z ##[endgroup]
2023-01-11T21:06:43.7689110Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_a41s99uk)
2023-01-11T21:06:43.7689386Z 
2023-01-11T21:06:43.7689589Z Running distributed/fsdp/test_fsdp_state_dict ... [2023-01-11 21:06:43.733651]
2023-01-11T21:06:43.7690117Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:06:43.734088]
2023-01-11T21:09:52.9377029Z 
2023-01-11T21:09:52.9377508Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_state_dict
2023-01-11T21:09:52.9378627Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_state_dict_azd3szu2)
2023-01-11T21:09:52.9379648Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict
2023-01-11T21:09:52.9379993Z 
2023-01-11T21:09:52.9380111Z Running tests...
2023-01-11T21:09:52.9380539Z ----------------------------------------------------------------------
2023-01-11T21:09:52.9412816Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9413533Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25520
2023-01-11T21:09:52.9414264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25521
2023-01-11T21:09:52.9414784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9415117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9415555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9421172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9422357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9423257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9424062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9424611Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9424964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9425324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9425871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9426448Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9426857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9427230Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9427504Z dist init r=0, world=2
2023-01-11T21:09:52.9427723Z dist init r=1, world=2
2023-01-11T21:09:52.9427932Z skip: Need at least 2 CUDA devices (1.865s)
2023-01-11T21:09:52.9428359Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9428891Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25548
2023-01-11T21:09:52.9429327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25549
2023-01-11T21:09:52.9429789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9430188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9430619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9430997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9431335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9431859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9432175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9432603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9432959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9433319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9433815Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9434403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9434793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9435154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9435426Z dist init r=1, world=2
2023-01-11T21:09:52.9435609Z dist init r=0, world=2
2023-01-11T21:09:52.9435806Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9436224Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9436739Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25576
2023-01-11T21:09:52.9437172Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25577
2023-01-11T21:09:52.9437619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9437945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9438378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9438725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9446815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9447621Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9448161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9448821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9449342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9449809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9450608Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9451641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9452255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9452821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9453266Z dist init r=1, world=2
2023-01-11T21:09:52.9453544Z dist init r=0, world=2
2023-01-11T21:09:52.9453852Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9454527Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9455348Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25604
2023-01-11T21:09:52.9455972Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25605
2023-01-11T21:09:52.9456686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9457216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9458080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9458612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9459284Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9460120Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9460641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9461343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9461860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9462369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9463159Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9464107Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9464680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9465170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9465522Z dist init r=1, world=2
2023-01-11T21:09:52.9465784Z dist init r=0, world=2
2023-01-11T21:09:52.9466074Z skip: Need at least 2 CUDA devices (1.615s)
2023-01-11T21:09:52.9466662Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9467447Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25632
2023-01-11T21:09:52.9468041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25633
2023-01-11T21:09:52.9468772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9469286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9469943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9470521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9471046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9471902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9472478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9473248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9473855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9474366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9475208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9476099Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9476848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9477457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9477880Z dist init r=1, world=2
2023-01-11T21:09:52.9478171Z dist init r=0, world=2
2023-01-11T21:09:52.9478547Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9479296Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9480245Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25660
2023-01-11T21:09:52.9480937Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25661
2023-01-11T21:09:52.9481745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9482300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9483076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9483681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9484288Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9485218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9485819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9486574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9487188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9487773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9488681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9489620Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9490255Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9490999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9491453Z dist init r=0, world=2
2023-01-11T21:09:52.9491769Z dist init r=1, world=2
2023-01-11T21:09:52.9492112Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9492835Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9493776Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25688
2023-01-11T21:09:52.9494466Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25689
2023-01-11T21:09:52.9495290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9495878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9496652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9497275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9497856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9498696Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9499295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9500134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9500745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9501322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9502355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9503295Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9503981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9504609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9505052Z dist init r=0, world=2
2023-01-11T21:09:52.9505346Z dist init r=1, world=2
2023-01-11T21:09:52.9505704Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9506452Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9507494Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25716
2023-01-11T21:09:52.9508175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25717
2023-01-11T21:09:52.9509014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9509609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9510370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9510979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9511577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9512385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9512961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9513740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9514336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9514897Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9515769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9516653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9517286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9517883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9518359Z dist init r=1, world=2
2023-01-11T21:09:52.9518692Z dist init r=0, world=2
2023-01-11T21:09:52.9519036Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9519763Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9520651Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25744
2023-01-11T21:09:52.9521332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25745
2023-01-11T21:09:52.9522168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9522743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9523520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9524246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9524803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9525677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9526276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9527016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9527608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9528204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9529093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9530108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9530797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9531556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9531989Z dist init r=0, world=2
2023-01-11T21:09:52.9532296Z dist init r=1, world=2
2023-01-11T21:09:52.9532656Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9533391Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9534286Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25772
2023-01-11T21:09:52.9535078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25773
2023-01-11T21:09:52.9535921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9536524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9537278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9537907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9538517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9539375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9540006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9540793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9541416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9542009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9542876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9543834Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9544514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9545123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9545564Z dist init r=1, world=2
2023-01-11T21:09:52.9545875Z dist init r=0, world=2
2023-01-11T21:09:52.9546230Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9547107Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9548026Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25800
2023-01-11T21:09:52.9548735Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25801
2023-01-11T21:09:52.9549598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9550165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9550939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9551554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9552262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9553152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9554066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9554661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9555443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9556060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9556659Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9557545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9558216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9558817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9559243Z dist init r=0, world=2
2023-01-11T21:09:52.9559536Z dist init r=1, world=2
2023-01-11T21:09:52.9559887Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9560607Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9561501Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25828
2023-01-11T21:09:52.9562213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25829
2023-01-11T21:09:52.9563067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9563636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9564391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9564988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9565587Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9566453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9567045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9567809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9568432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9569141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9570044Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9571172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9571837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9573808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9574245Z dist init r=1, world=2
2023-01-11T21:09:52.9574556Z dist init r=0, world=2
2023-01-11T21:09:52.9574913Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9575743Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9576698Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25856
2023-01-11T21:09:52.9577400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25857
2023-01-11T21:09:52.9578252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9578798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9579642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9580256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9580814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9581645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9582226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9582994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9583559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9584090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9584951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9585882Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9586529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9587172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9587626Z dist init r=1, world=2
2023-01-11T21:09:52.9587944Z dist init r=0, world=2
2023-01-11T21:09:52.9588280Z skip: Need at least 2 CUDA devices (1.615s)
2023-01-11T21:09:52.9588935Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9589816Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25884
2023-01-11T21:09:52.9590507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25885
2023-01-11T21:09:52.9591355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9591897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9592786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9593369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9593939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9594774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9595352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9596073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9596661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9597233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9598195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9599117Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9599799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9600415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9600860Z dist init r=1, world=2
2023-01-11T21:09:52.9601158Z dist init r=0, world=2
2023-01-11T21:09:52.9601518Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9602219Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9603137Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25912
2023-01-11T21:09:52.9603833Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25913
2023-01-11T21:09:52.9604676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9605239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9606027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9606659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9607242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9608090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9608671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9609450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9610056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9610655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9611722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9612658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9613323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9613929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9614345Z dist init r=1, world=2
2023-01-11T21:09:52.9614769Z dist init r=0, world=2
2023-01-11T21:09:52.9615144Z skip: Need at least 2 CUDA devices (1.617s)
2023-01-11T21:09:52.9615854Z   test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9616743Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25940
2023-01-11T21:09:52.9617393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25941
2023-01-11T21:09:52.9618221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9618810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9619694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9620449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9621042Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9621898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9622480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9623265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9623879Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9624489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9625344Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9626264Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9626944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9627545Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9627980Z dist init r=0, world=2
2023-01-11T21:09:52.9628296Z dist init r=1, world=2
2023-01-11T21:09:52.9628642Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9629352Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9630283Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25968
2023-01-11T21:09:52.9630978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25969
2023-01-11T21:09:52.9631833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9632404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9633185Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9633757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9634312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9635163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9635744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9636527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9637219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9637821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9638708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9639644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9640334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9640945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9641399Z dist init r=1, world=2
2023-01-11T21:09:52.9641700Z dist init r=0, world=2
2023-01-11T21:09:52.9642068Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9642896Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9643809Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25996
2023-01-11T21:09:52.9644461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25997
2023-01-11T21:09:52.9645310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9645879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9646649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9647243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9647828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9648689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9649253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9650031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9650651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9651441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9652353Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9653299Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9653962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9654556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9654984Z dist init r=0, world=2
2023-01-11T21:09:52.9655288Z dist init r=1, world=2
2023-01-11T21:09:52.9655655Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9656366Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9657264Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26024
2023-01-11T21:09:52.9657965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26025
2023-01-11T21:09:52.9658811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9659608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9660374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9660977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9661548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9662399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9662995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9663768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9664233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9664678Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9665192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9665718Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9666093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9666438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9666701Z dist init r=1, world=2
2023-01-11T21:09:52.9666871Z dist init r=0, world=2
2023-01-11T21:09:52.9667077Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9667502Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9668024Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26052
2023-01-11T21:09:52.9668397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26053
2023-01-11T21:09:52.9668852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9669184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9669610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9669938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9670271Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9670739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9671050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9671473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9671814Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9672144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9672621Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9673137Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9673517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9673924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9674167Z dist init r=0, world=2
2023-01-11T21:09:52.9674350Z dist init r=1, world=2
2023-01-11T21:09:52.9674559Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9674964Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9675483Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26080
2023-01-11T21:09:52.9675872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26081
2023-01-11T21:09:52.9676322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9676635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9677092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9677432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9677751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9678219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9678539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9678959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9679284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9679613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9680107Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9680619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9680986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9681330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9681586Z dist init r=0, world=2
2023-01-11T21:09:52.9681757Z dist init r=1, world=2
2023-01-11T21:09:52.9681960Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9682374Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9682887Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26108
2023-01-11T21:09:52.9683261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26109
2023-01-11T21:09:52.9683708Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9684031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9684491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9684818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9685150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9685614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9685924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9686375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9686713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9687041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9687518Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9688028Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9688409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9688750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9688993Z dist init r=0, world=2
2023-01-11T21:09:52.9689199Z dist init r=1, world=2
2023-01-11T21:09:52.9689405Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9689808Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9690322Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26136
2023-01-11T21:09:52.9690705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26137
2023-01-11T21:09:52.9691350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9691692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9692116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9692465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9692786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9693249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9693572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9693992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9694316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9694645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9695135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9695655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9696024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9696365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9696620Z dist init r=1, world=2
2023-01-11T21:09:52.9696783Z dist init r=0, world=2
2023-01-11T21:09:52.9696990Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9697403Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9697915Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26164
2023-01-11T21:09:52.9698289Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26165
2023-01-11T21:09:52.9698809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9699138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9699636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9699964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9700299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9700769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9701078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9701545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9701889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9702217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9702695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9703209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9703591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9703938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9704186Z dist init r=1, world=2
2023-01-11T21:09:52.9704366Z dist init r=0, world=2
2023-01-11T21:09:52.9704574Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9704984Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9705499Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26192
2023-01-11T21:09:52.9705885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26193
2023-01-11T21:09:52.9706333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9706646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9707066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9707405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9707726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9708191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9708512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9708935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9709257Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9709589Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9710082Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9710595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9711006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9711350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9711606Z dist init r=1, world=2
2023-01-11T21:09:52.9711773Z dist init r=0, world=2
2023-01-11T21:09:52.9711980Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9712393Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9712912Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26220
2023-01-11T21:09:52.9713286Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26221
2023-01-11T21:09:52.9713768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9714097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9714515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9714858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9715193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9715660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9715972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9716398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9716740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9717074Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9717551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9718064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9718445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9718777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9719037Z dist init r=0, world=2
2023-01-11T21:09:52.9719220Z dist init r=1, world=2
2023-01-11T21:09:52.9719429Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9719836Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9720357Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26248
2023-01-11T21:09:52.9720745Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26249
2023-01-11T21:09:52.9721197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9721510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9721935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9722277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9722597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9723072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9723433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9723861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9724185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9724516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9725003Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9725523Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9725893Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9726266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9726527Z dist init r=0, world=2
2023-01-11T21:09:52.9726695Z dist init r=1, world=2
2023-01-11T21:09:52.9726898Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9727311Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9727822Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26276
2023-01-11T21:09:52.9728195Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26277
2023-01-11T21:09:52.9728647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9728973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9729388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9729726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9730061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9730527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9730977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9731414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9731755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9732089Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9732571Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9733088Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9733468Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9733798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9734053Z dist init r=1, world=2
2023-01-11T21:09:52.9734232Z dist init r=0, world=2
2023-01-11T21:09:52.9734438Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9734844Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9735362Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26304
2023-01-11T21:09:52.9735811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26305
2023-01-11T21:09:52.9736260Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9736573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9737001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9737345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9737665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9738129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9738491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9738912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9739237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9739646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9740140Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9740641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9741025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9741376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9741634Z dist init r=1, world=2
2023-01-11T21:09:52.9741806Z dist init r=0, world=2
2023-01-11T21:09:52.9742036Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9742452Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9742959Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26332
2023-01-11T21:09:52.9743335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26333
2023-01-11T21:09:52.9743787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9744119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9744535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9744882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9745212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9745677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9745988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9746408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9746749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9747067Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9747555Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9748114Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9748494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9748824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9749082Z dist init r=1, world=2
2023-01-11T21:09:52.9749261Z dist init r=0, world=2
2023-01-11T21:09:52.9749457Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9749871Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9750389Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26360
2023-01-11T21:09:52.9750804Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26361
2023-01-11T21:09:52.9751245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9751574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9751992Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9752334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9752653Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9753116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9753438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9753860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9754188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9754517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9755003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9755507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9755895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9756243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9756497Z dist init r=1, world=2
2023-01-11T21:09:52.9756662Z dist init r=0, world=2
2023-01-11T21:09:52.9756866Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9757279Z   test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9757783Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26388
2023-01-11T21:09:52.9758167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26389
2023-01-11T21:09:52.9758617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9758944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9759360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9759700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9760032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9760549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9760859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9761278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9761618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9761937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9762426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9762944Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9763363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9763692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9763944Z dist init r=1, world=2
2023-01-11T21:09:52.9764127Z dist init r=0, world=2
2023-01-11T21:09:52.9764322Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9764733Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9765241Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26416
2023-01-11T21:09:52.9765629Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26417
2023-01-11T21:09:52.9766065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9766398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9766821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9767168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9767487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9767958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9768284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9768710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9769047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9769388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9769881Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9770388Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9770778Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9771274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9771537Z dist init r=0, world=2
2023-01-11T21:09:52.9771707Z dist init r=1, world=2
2023-01-11T21:09:52.9771916Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9772333Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9772887Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26444
2023-01-11T21:09:52.9773278Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26445
2023-01-11T21:09:52.9773741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9774072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9774486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9774830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9775165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9775673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9775992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9776422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9776765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9777088Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9777590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9778124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9778508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9778847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9779106Z dist init r=0, world=2
2023-01-11T21:09:52.9779290Z dist init r=1, world=2
2023-01-11T21:09:52.9779566Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9779976Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9780488Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26472
2023-01-11T21:09:52.9780881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26473
2023-01-11T21:09:52.9781327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9781658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9782085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9782428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9782744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9783215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9783540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9783948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9784286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9784621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9785119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9785660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9786044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9786389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9786648Z dist init r=0, world=2
2023-01-11T21:09:52.9786817Z dist init r=1, world=2
2023-01-11T21:09:52.9787021Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9787436Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9787999Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26500
2023-01-11T21:09:52.9788395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26501
2023-01-11T21:09:52.9788850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9789177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9789591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9789933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9790268Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9790736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9791045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9791473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9791811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9792126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9792617Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9793134Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9793519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9793850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9794101Z dist init r=1, world=2
2023-01-11T21:09:52.9794278Z dist init r=0, world=2
2023-01-11T21:09:52.9794475Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9794881Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9795388Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26528
2023-01-11T21:09:52.9795779Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26529
2023-01-11T21:09:52.9796221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9796545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9796972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9797325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9797691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9798154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9798478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9798896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9799237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9799570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9800064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9800599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9800989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9801332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9801591Z dist init r=1, world=2
2023-01-11T21:09:52.9801761Z dist init r=0, world=2
2023-01-11T21:09:52.9801970Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9802385Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9802878Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26556
2023-01-11T21:09:52.9803270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26557
2023-01-11T21:09:52.9803734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9804059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9804473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9804813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9805146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9805597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9805921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9806346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9806691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9807006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9807496Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9808017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9808402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9808735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9808987Z dist init r=0, world=2
2023-01-11T21:09:52.9809169Z dist init r=1, world=2
2023-01-11T21:09:52.9809362Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9809775Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9810325Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26584
2023-01-11T21:09:52.9810713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26585
2023-01-11T21:09:52.9811480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9811827Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9812262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9812608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9812995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9813470Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9813797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9814215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9814557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9814891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9815391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9815897Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9816280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9816629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9816879Z dist init r=0, world=2
2023-01-11T21:09:52.9817070Z dist init r=1, world=2
2023-01-11T21:09:52.9817276Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9817684Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=False)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9818177Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26612
2023-01-11T21:09:52.9818571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26613
2023-01-11T21:09:52.9819029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9819366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9819854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9820197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9820532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9820989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9821321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9821743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9822078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9822398Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9822962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9823482Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9823862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9824191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9824445Z dist init r=0, world=2
2023-01-11T21:09:52.9824629Z dist init r=1, world=2
2023-01-11T21:09:52.9824826Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9825238Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9825778Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26640
2023-01-11T21:09:52.9826171Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26641
2023-01-11T21:09:52.9826616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9826948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9827385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9827717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9828054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9828521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9828856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9829270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9829618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9829955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9830452Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9830962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9831350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9831704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9831964Z dist init r=0, world=2
2023-01-11T21:09:52.9832149Z dist init r=1, world=2
2023-01-11T21:09:52.9832359Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9832767Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9833266Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26668
2023-01-11T21:09:52.9833661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26669
2023-01-11T21:09:52.9834127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9834456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9834877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9835265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9835598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9836049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9836373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9836799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9837149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9837469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9837996Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9838520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9838910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9839244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9839505Z dist init r=1, world=2
2023-01-11T21:09:52.9839686Z dist init r=0, world=2
2023-01-11T21:09:52.9839881Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9840296Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9840814Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26696
2023-01-11T21:09:52.9841214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26697
2023-01-11T21:09:52.9841660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9841984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9842405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9842734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9843060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9843522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9843849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9844262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9844606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9844944Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9845437Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9845941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9846323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9846674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9846919Z dist init r=1, world=2
2023-01-11T21:09:52.9847096Z dist init r=0, world=2
2023-01-11T21:09:52.9847305Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9847777Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9848274Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26724
2023-01-11T21:09:52.9848671Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26725
2023-01-11T21:09:52.9849126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9849440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9849863Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9850205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9850573Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9851201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9851533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9851957Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9852297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9852614Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9853108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9853625Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9853994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9854337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9854587Z dist init r=0, world=2
2023-01-11T21:09:52.9854767Z dist init r=1, world=2
2023-01-11T21:09:52.9854961Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9855366Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9855876Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26752
2023-01-11T21:09:52.9856260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26753
2023-01-11T21:09:52.9856704Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9857032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9857458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9857785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9858119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9858581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9858901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9859316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9859736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9860138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9860619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9861140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9861523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9861866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9862112Z dist init r=1, world=2
2023-01-11T21:09:52.9862290Z dist init r=0, world=2
2023-01-11T21:09:52.9862494Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9862929Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9863442Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26780
2023-01-11T21:09:52.9863823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26781
2023-01-11T21:09:52.9864276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9864589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9865017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9865360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9865692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9866154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9866482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9866904Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9867241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9867558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9868051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9868567Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9868936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9869283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9869535Z dist init r=0, world=2
2023-01-11T21:09:52.9869713Z dist init r=1, world=2
2023-01-11T21:09:52.9869907Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9870314Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9870819Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26808
2023-01-11T21:09:52.9871193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26809
2023-01-11T21:09:52.9871643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9871968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9872436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9872763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9873099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9873566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9873890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9874306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9874643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9874978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9875497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9876016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9876400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9876744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9876989Z dist init r=1, world=2
2023-01-11T21:09:52.9877169Z dist init r=0, world=2
2023-01-11T21:09:52.9877374Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9877768Z   test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload_CPUOffload(offload_params=True)_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9878278Z Tests that we can save a state_dict and load it into a blank model ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26836
2023-01-11T21:09:52.9878667Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26837
2023-01-11T21:09:52.9879119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9879430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9879851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9880196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9880532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9880984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9881314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9881743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9882076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9882412Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9883107Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9883857Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9884373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9884849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9885205Z dist init r=1, world=2
2023-01-11T21:09:52.9885542Z dist init r=0, world=2
2023-01-11T21:09:52.9885837Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9886427Z   test_fsdp_state_dict_keys_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26864
2023-01-11T21:09:52.9887079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26865
2023-01-11T21:09:52.9887851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9888429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9889171Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9889747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9890303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9891413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9891943Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9892510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9892857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9893193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9893689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9894204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9894592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9894940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9895202Z dist init r=1, world=2
2023-01-11T21:09:52.9895374Z dist init r=0, world=2
2023-01-11T21:09:52.9895579Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:09:52.9895961Z   test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26892
2023-01-11T21:09:52.9896368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26893
2023-01-11T21:09:52.9896823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9897149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9897570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9897905Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9898273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9899020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9899359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9899853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9900202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9900537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9901017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9901641Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9902033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9902379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9902624Z dist init r=0, world=2
2023-01-11T21:09:52.9902806Z dist init r=1, world=2
2023-01-11T21:09:52.9903015Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:09:52.9903380Z   test_fsdp_state_dict_keys_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26920
2023-01-11T21:09:52.9903796Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26921
2023-01-11T21:09:52.9904245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9904597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9905029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9905366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9905697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9906146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9906467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9906886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9907226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9907543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9908037Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9908554Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9908922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9909259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9909509Z dist init r=1, world=2
2023-01-11T21:09:52.9909685Z dist init r=0, world=2
2023-01-11T21:09:52.9909877Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:09:52.9910257Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9910861Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26948
2023-01-11T21:09:52.9911259Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26949
2023-01-11T21:09:52.9911522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9911644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9911928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9912061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9912238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9912508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9912668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9912946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9913064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9913237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9913535Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9913830Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9913992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9914153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9914231Z dist init r=0, world=2
2023-01-11T21:09:52.9914335Z dist init r=1, world=2
2023-01-11T21:09:52.9914426Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:09:52.9914678Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9915015Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26976
2023-01-11T21:09:52.9915169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26977
2023-01-11T21:09:52.9915438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9915558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9915833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9915969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9916145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9916402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9916521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9916798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9916931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9917106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9917403Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9917702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9917867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9918027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9918088Z dist init r=0, world=2
2023-01-11T21:09:52.9918159Z dist init r=1, world=2
2023-01-11T21:09:52.9918261Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9918500Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9918835Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27004
2023-01-11T21:09:52.9918988Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27005
2023-01-11T21:09:52.9919291Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9919411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9919688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9919809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9919982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9920254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9920373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9920650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9920812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9920988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9921286Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9921569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9921731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9921891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9921965Z dist init r=0, world=2
2023-01-11T21:09:52.9922036Z dist init r=1, world=2
2023-01-11T21:09:52.9922138Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9922378Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9922714Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27032
2023-01-11T21:09:52.9922868Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27033
2023-01-11T21:09:52.9923126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9923247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9923525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9923656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9923832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9924102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9924220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9924496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9924614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9924786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9925082Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9925375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9925536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9925743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9925821Z dist init r=1, world=2
2023-01-11T21:09:52.9925892Z dist init r=0, world=2
2023-01-11T21:09:52.9925982Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9926221Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9926559Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27060
2023-01-11T21:09:52.9926712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27061
2023-01-11T21:09:52.9926984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9927105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9927417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9927559Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9927734Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9927993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9928114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9928390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9928521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9928695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9928997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9929289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9929453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9929612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9929674Z dist init r=0, world=2
2023-01-11T21:09:52.9929745Z dist init r=1, world=2
2023-01-11T21:09:52.9929845Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9930081Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9930417Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27088
2023-01-11T21:09:52.9930571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27089
2023-01-11T21:09:52.9930991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9931128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9931409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9931530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9931704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9931975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9932096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9932431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9932564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9932738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9933034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9933315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9933478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9933640Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9933715Z dist init r=0, world=2
2023-01-11T21:09:52.9933788Z dist init r=1, world=2
2023-01-11T21:09:52.9933933Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9934189Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9934524Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27116
2023-01-11T21:09:52.9934683Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27117
2023-01-11T21:09:52.9934943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9935062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9935342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9935475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9935651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9935920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9936041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9936316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9936435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9936610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9936906Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9937204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9937368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9937529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9937602Z dist init r=0, world=2
2023-01-11T21:09:52.9937674Z dist init r=1, world=2
2023-01-11T21:09:52.9937763Z skip: Need at least 2 CUDA devices (1.714s)
2023-01-11T21:09:52.9938012Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9938345Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27144
2023-01-11T21:09:52.9938499Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27145
2023-01-11T21:09:52.9938773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9938925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9939204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9939338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9939584Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9939882Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9940140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9940264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9940579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9940718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9940894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9941200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9941364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9941527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9941590Z dist init r=0, world=2
2023-01-11T21:09:52.9941663Z dist init r=1, world=2
2023-01-11T21:09:52.9941768Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9942014Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9942349Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27172
2023-01-11T21:09:52.9942506Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27173
2023-01-11T21:09:52.9942781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9942902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9943181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9943301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9943482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9943752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9943877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9944155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9944288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9944464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9944761Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9945045Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9945208Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9945373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9945484Z dist init r=1, world=2
2023-01-11T21:09:52.9945556Z dist init r=0, world=2
2023-01-11T21:09:52.9945660Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:09:52.9945903Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9946242Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27200
2023-01-11T21:09:52.9946398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27201
2023-01-11T21:09:52.9946658Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9946781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9947089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9947225Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9947401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9947670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9947792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9948068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9948188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9948365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9948664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9948961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9949125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9949290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9949367Z dist init r=0, world=2
2023-01-11T21:09:52.9949441Z dist init r=1, world=2
2023-01-11T21:09:52.9949545Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9949779Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9950116Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27228
2023-01-11T21:09:52.9950277Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27229
2023-01-11T21:09:52.9950551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9950676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9950952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9951085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9951258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9951524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9951634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9951911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9952112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9952287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9952581Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9952874Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9953037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9953200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9953261Z dist init r=1, world=2
2023-01-11T21:09:52.9953338Z dist init r=0, world=2
2023-01-11T21:09:52.9953443Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:09:52.9953717Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9954055Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27256
2023-01-11T21:09:52.9954212Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27257
2023-01-11T21:09:52.9954486Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9954608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9954886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9955006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9955185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9955456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9955576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9955850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9955984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9956158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9956453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9956735Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9956901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9957063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9957140Z dist init r=1, world=2
2023-01-11T21:09:52.9957211Z dist init r=0, world=2
2023-01-11T21:09:52.9957311Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9957543Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9957879Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27284
2023-01-11T21:09:52.9958034Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27285
2023-01-11T21:09:52.9958295Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9958446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9958724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9958857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9959029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9959293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9959413Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9959686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9959804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9960003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9960303Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9960599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9960760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9960919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9960992Z dist init r=1, world=2
2023-01-11T21:09:52.9961064Z dist init r=0, world=2
2023-01-11T21:09:52.9961166Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9961384Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9961722Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27312
2023-01-11T21:09:52.9961878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27313
2023-01-11T21:09:52.9962147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9962270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9962545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9962677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9962848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9963114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9963227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9963501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9963633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9963802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9964096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9964389Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9964551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9964714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9964777Z dist init r=1, world=2
2023-01-11T21:09:52.9964884Z dist init r=0, world=2
2023-01-11T21:09:52.9964985Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9965217Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9965558Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27340
2023-01-11T21:09:52.9965712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27341
2023-01-11T21:09:52.9965983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9966103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9966380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9966535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9966711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9966978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9967097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9967372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9967504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9967677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9967971Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9968257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9968420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9968580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9968657Z dist init r=1, world=2
2023-01-11T21:09:52.9968729Z dist init r=0, world=2
2023-01-11T21:09:52.9968832Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9969063Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9969394Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27368
2023-01-11T21:09:52.9969549Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27369
2023-01-11T21:09:52.9969813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9969935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9970215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9970348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9970521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9970786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9971072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9971356Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9971480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9971720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9972018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9972315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9972479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9972642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9972719Z dist init r=0, world=2
2023-01-11T21:09:52.9972792Z dist init r=1, world=2
2023-01-11T21:09:52.9972883Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:09:52.9973172Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9973518Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27396
2023-01-11T21:09:52.9973674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27397
2023-01-11T21:09:52.9973950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9974070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9974347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9974481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9974657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9974916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9975037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9975313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9975444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9975616Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9975914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9976207Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9976370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9976537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9976598Z dist init r=1, world=2
2023-01-11T21:09:52.9976670Z dist init r=0, world=2
2023-01-11T21:09:52.9976774Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9977024Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9977361Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27424
2023-01-11T21:09:52.9977516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27425
2023-01-11T21:09:52.9977786Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9977907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9978218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9978339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9978514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9978779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9978900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9979175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9979307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9979546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9979885Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9980169Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9980337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9980499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9980575Z dist init r=1, world=2
2023-01-11T21:09:52.9980648Z dist init r=0, world=2
2023-01-11T21:09:52.9980750Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9980985Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9981329Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27452
2023-01-11T21:09:52.9981489Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27453
2023-01-11T21:09:52.9981748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9981871Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9982154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9982287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9982459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9982732Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9982853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9983131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9983254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9983427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9983720Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9984015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9984177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9984337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9984411Z dist init r=0, world=2
2023-01-11T21:09:52.9984483Z dist init r=1, world=2
2023-01-11T21:09:52.9984575Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9984837Z   test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9985173Z Tests saving the state dict, zeroing a target model's parameters, and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27480
2023-01-11T21:09:52.9985328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27481
2023-01-11T21:09:52.9985598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9985719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9985997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9986129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9986331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9986585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9986706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9986979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9987108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9987279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9987577Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9987872Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9988040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9988199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9988261Z dist init r=1, world=2
2023-01-11T21:09:52.9988331Z dist init r=0, world=2
2023-01-11T21:09:52.9988433Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9988672Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9988901Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27508
2023-01-11T21:09:52.9989053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27509
2023-01-11T21:09:52.9989324Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9989446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9989724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9989844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9990019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9990285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9990404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9990680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9990809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9990983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9991308Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9991590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9991750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9991914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9991988Z dist init r=0, world=2
2023-01-11T21:09:52.9992060Z dist init r=1, world=2
2023-01-11T21:09:52.9992161Z skip: Need at least 2 CUDA devices (1.615s)
2023-01-11T21:09:52.9992399Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9992662Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27536
2023-01-11T21:09:52.9992819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27537
2023-01-11T21:09:52.9993080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9993201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9993478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9993610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9993783Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9994051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9994178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9994453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9994572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9994743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9995040Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9995332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9995492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9995653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9995731Z dist init r=0, world=2
2023-01-11T21:09:52.9995803Z dist init r=1, world=2
2023-01-11T21:09:52.9995893Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9996128Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:52.9996361Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27564
2023-01-11T21:09:52.9996516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27565
2023-01-11T21:09:52.9996789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9996914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9997195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9997358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9997535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:52.9997791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:52.9997909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:52.9998186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:52.9998319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:52.9998497Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:52.9998796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9999120Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:52.9999287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:52.9999451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:52.9999514Z dist init r=0, world=2
2023-01-11T21:09:52.9999588Z dist init r=1, world=2
2023-01-11T21:09:52.9999693Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:52.9999928Z   test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0000162Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27592
2023-01-11T21:09:53.0018406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27593
2023-01-11T21:09:53.0018821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0018953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0019230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0019368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0019639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0019918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0020040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0020327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0020465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0020644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0020935Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0021234Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0021399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0021564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0021641Z dist init r=0, world=2
2023-01-11T21:09:53.0021713Z dist init r=1, world=2
2023-01-11T21:09:53.0021817Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0022060Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0022420Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27620
2023-01-11T21:09:53.0022563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27621
2023-01-11T21:09:53.0022839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0022959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0023236Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0023373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0023586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0023858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0023978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0024244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0024374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0024542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0024840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0025133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0025302Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0025467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0025541Z dist init r=1, world=2
2023-01-11T21:09:53.0025611Z dist init r=0, world=2
2023-01-11T21:09:53.0025703Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0025947Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0026178Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27648
2023-01-11T21:09:53.0026335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27649
2023-01-11T21:09:53.0026609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0026732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0027020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0027159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0027334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0027595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0027717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0027996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0028130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0028305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0028653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0028949Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0029114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0029263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0029333Z dist init r=1, world=2
2023-01-11T21:09:53.0029400Z dist init r=0, world=2
2023-01-11T21:09:53.0029501Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0029734Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0029993Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27676
2023-01-11T21:09:53.0030150Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27677
2023-01-11T21:09:53.0030423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0030546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0030812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0030945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0031121Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0031389Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0031506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0031788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0031917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0032086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0032372Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0032669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0032831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0032992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0033066Z dist init r=0, world=2
2023-01-11T21:09:53.0033143Z dist init r=1, world=2
2023-01-11T21:09:53.0033246Z skip: Need at least 2 CUDA devices (1.615s)
2023-01-11T21:09:53.0033484Z   test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0033716Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27704
2023-01-11T21:09:53.0033859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27705
2023-01-11T21:09:53.0034131Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0034254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0034531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0034698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0034873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0035142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0035263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0035526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0035659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0035832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0036127Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0036458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0036623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0036785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0036859Z dist init r=0, world=2
2023-01-11T21:09:53.0036930Z dist init r=1, world=2
2023-01-11T21:09:53.0037019Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0037253Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0037485Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27732
2023-01-11T21:09:53.0037639Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27733
2023-01-11T21:09:53.0037916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0038038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0038316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0038450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0038624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0038880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0039002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0039281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0039418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0039595Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0039890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0040187Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0040349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0040499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0040575Z dist init r=1, world=2
2023-01-11T21:09:53.0040649Z dist init r=0, world=2
2023-01-11T21:09:53.0040751Z skip: Need at least 2 CUDA devices (1.615s)
2023-01-11T21:09:53.0040983Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0041246Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27760
2023-01-11T21:09:53.0041401Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27761
2023-01-11T21:09:53.0041674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0041794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0042062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0042194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0042367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0042666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0042791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0043073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0043206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0043383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0043667Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0043962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0044128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0044295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0044368Z dist init r=0, world=2
2023-01-11T21:09:53.0044439Z dist init r=1, world=2
2023-01-11T21:09:53.0044541Z skip: Need at least 2 CUDA devices (1.615s)
2023-01-11T21:09:53.0044770Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0045002Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27788
2023-01-11T21:09:53.0045143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27789
2023-01-11T21:09:53.0045414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0045534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0045816Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0045951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0046125Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0046417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0046685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0046806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0047070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0047203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0047383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0047708Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0047873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0048035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0048110Z dist init r=0, world=2
2023-01-11T21:09:53.0048182Z dist init r=1, world=2
2023-01-11T21:09:53.0048273Z skip: Need at least 2 CUDA devices (1.615s)
2023-01-11T21:09:53.0048502Z   test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0048734Z Test that saving after some training results in params being updated as ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27816
2023-01-11T21:09:53.0048919Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27817
2023-01-11T21:09:53.0049191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0049313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0049591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0049724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0049886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0050157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0050279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0050560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0050696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0051052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0051402Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0051699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0051864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0052011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0052090Z dist init r=0, world=2
2023-01-11T21:09:53.0052165Z dist init r=1, world=2
2023-01-11T21:09:53.0052270Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0052505Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0052830Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27844
2023-01-11T21:09:53.0052986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27845
2023-01-11T21:09:53.0053259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0053381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0053648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0053784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0053961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0054297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0054419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0054697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0054832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0055009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0055292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0055588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0055793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0055955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0056032Z dist init r=1, world=2
2023-01-11T21:09:53.0056106Z dist init r=0, world=2
2023-01-11T21:09:53.0056210Z skip: Need at least 2 CUDA devices (1.615s)
2023-01-11T21:09:53.0056439Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0056764Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27872
2023-01-11T21:09:53.0056907Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27873
2023-01-11T21:09:53.0057180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0057305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0057583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0057717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0057894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0058167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0058289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0058556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0058691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0058869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0059171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0059536Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0059705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0059866Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0059942Z dist init r=1, world=2
2023-01-11T21:09:53.0060016Z dist init r=0, world=2
2023-01-11T21:09:53.0060106Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0060335Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0060661Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27900
2023-01-11T21:09:53.0060851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27901
2023-01-11T21:09:53.0061127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0061250Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0061529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0061662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0061822Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0062091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0062278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0062558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0062691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0062869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0063168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0063464Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0063629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0063777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0063854Z dist init r=0, world=2
2023-01-11T21:09:53.0063931Z dist init r=1, world=2
2023-01-11T21:09:53.0064035Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0064263Z   test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0064585Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27928
2023-01-11T21:09:53.0064740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27929
2023-01-11T21:09:53.0065012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0065121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0065401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0065537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0065716Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0065984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0066106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0066384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0066516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0066690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0066970Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0067269Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0067467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0067626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0067700Z dist init r=0, world=2
2023-01-11T21:09:53.0067771Z dist init r=1, world=2
2023-01-11T21:09:53.0067874Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0068093Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0068396Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27956
2023-01-11T21:09:53.0068551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27957
2023-01-11T21:09:53.0068847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0068971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0069250Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0069387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0069559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0069828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0069948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0070212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0070343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0070521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0070817Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0071111Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0071272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0071432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0071506Z dist init r=1, world=2
2023-01-11T21:09:53.0071565Z dist init r=0, world=2
2023-01-11T21:09:53.0071668Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0071885Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0072208Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27984
2023-01-11T21:09:53.0072368Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27985
2023-01-11T21:09:53.0072636Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0072756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0073033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0073168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0073330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0073599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0073760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0074040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0074172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0074345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0074644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0074942Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0075104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0075280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0075357Z dist init r=0, world=2
2023-01-11T21:09:53.0075429Z dist init r=1, world=2
2023-01-11T21:09:53.0075532Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0075751Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0076073Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28012
2023-01-11T21:09:53.0076225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28013
2023-01-11T21:09:53.0076490Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0076598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0076877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0077007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0077176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0077443Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0077560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0077835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0077969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0078139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0078435Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0078729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0078884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0079046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0079108Z dist init r=0, world=2
2023-01-11T21:09:53.0079174Z dist init r=1, world=2
2023-01-11T21:09:53.0079277Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0079489Z   test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0079808Z Tests that FSDP's state_dict can be loaded into a local model. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28040
2023-01-11T21:09:53.0079956Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28041
2023-01-11T21:09:53.0080258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0080372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0080635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0080767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0080933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0081195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0081317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0081588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0081745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0081918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0082202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0082489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0082645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0082808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0082875Z dist init r=1, world=2
2023-01-11T21:09:53.0082947Z dist init r=0, world=2
2023-01-11T21:09:53.0083044Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0083208Z   test_state_dict_rank0_offload_save_load_flow_use_orig_params_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0083436Z Tests saving a model checkpoint only on rank 0 and loading it only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28068
2023-01-11T21:09:53.0083581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28069
2023-01-11T21:09:53.0083850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0083970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0084244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0084372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0084540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0084812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0084931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0085199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0085331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0085499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0085787Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0086080Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0086236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0086397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0086496Z dist init r=0, world=2
2023-01-11T21:09:53.0086564Z dist init r=1, world=2
2023-01-11T21:09:53.0086655Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0086824Z   test_state_dict_rank0_offload_save_load_flow_use_orig_params_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0087041Z Tests saving a model checkpoint only on rank 0 and loading it only ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28096
2023-01-11T21:09:53.0087189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28097
2023-01-11T21:09:53.0087455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0087578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0087877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0088009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0088169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0088436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0088549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0088819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0088946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0089120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0089408Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0089705Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0089868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0090013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0090081Z dist init r=1, world=2
2023-01-11T21:09:53.0090148Z dist init r=0, world=2
2023-01-11T21:09:53.0090251Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0090499Z   test_state_dict_save_load_flow_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28124
2023-01-11T21:09:53.0090649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28125
2023-01-11T21:09:53.0091082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0091212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0091485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0091613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0091780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0092051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0092173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0092452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0092588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0092765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0093124Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0093408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0093572Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0093733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0093801Z dist init r=1, world=2
2023-01-11T21:09:53.0093874Z dist init r=0, world=2
2023-01-11T21:09:53.0093971Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:09:53.0094227Z   test_state_dict_save_load_flow_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28152
2023-01-11T21:09:53.0094415Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28153
2023-01-11T21:09:53.0094676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0094791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0095072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0095199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0095367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0095639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0095756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0096033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0096167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0096328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0096619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0096908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0097067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0097223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0097291Z dist init r=1, world=2
2023-01-11T21:09:53.0097362Z dist init r=0, world=2
2023-01-11T21:09:53.0097458Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:09:53.0097695Z   test_state_dict_save_load_flow_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28180
2023-01-11T21:09:53.0097844Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28181
2023-01-11T21:09:53.0098114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0098231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0098502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0098634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0098802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0099070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0099215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0099535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0099668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0099838Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0100127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0100416Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0100571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0100765Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0100837Z dist init r=1, world=2
2023-01-11T21:09:53.0100897Z dist init r=0, world=2
2023-01-11T21:09:53.0100993Z skip: Need at least 2 CUDA devices (1.612s)
2023-01-11T21:09:53.0101261Z   test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28208
2023-01-11T21:09:53.0101410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28209
2023-01-11T21:09:53.0101685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0101800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0102072Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0102200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0102374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0102632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0102747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0103019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0103144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0103312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0103601Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0103898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0104059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0104209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0104286Z dist init r=1, world=2
2023-01-11T21:09:53.0104359Z dist init r=0, world=2
2023-01-11T21:09:53.0104464Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0104736Z   test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28236
2023-01-11T21:09:53.0104892Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28237
2023-01-11T21:09:53.0105167Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0105287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0105600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0105722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0105900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0106169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0106290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0106568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0106704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0106879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0107205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0107491Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0107655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0107819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0107895Z dist init r=1, world=2
2023-01-11T21:09:53.0107968Z dist init r=0, world=2
2023-01-11T21:09:53.0108072Z skip: Need at least 2 CUDA devices (1.615s)
2023-01-11T21:09:53.0108334Z   test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28264
2023-01-11T21:09:53.0108490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28265
2023-01-11T21:09:53.0108767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0108877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0109158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0109293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0109467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0109739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0109861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0110138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0110277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0110439Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0110738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0111034Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0111196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0111357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0111434Z dist init r=0, world=2
2023-01-11T21:09:53.0111507Z dist init r=1, world=2
2023-01-11T21:09:53.0111612Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0111815Z   test_state_dict_type (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28292
2023-01-11T21:09:53.0112001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28293
2023-01-11T21:09:53.0112275Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0112395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0112675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0112809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0112983Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0113257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0113380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0113675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0113811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0113984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0114281Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0114573Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0114727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0114879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0114946Z dist init r=1, world=2
2023-01-11T21:09:53.0115005Z dist init r=0, world=2
2023-01-11T21:09:53.0115112Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:09:53.0115406Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28320
2023-01-11T21:09:53.0115558Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28321
2023-01-11T21:09:53.0115827Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0115952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0116228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0116359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0116533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0116788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0116907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0117184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0117318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0117491Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0117785Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0118077Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0118241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0118452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0118514Z dist init r=0, world=2
2023-01-11T21:09:53.0118585Z dist init r=1, world=2
2023-01-11T21:09:53.0118686Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0118978Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28348
2023-01-11T21:09:53.0119130Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28349
2023-01-11T21:09:53.0119403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0119524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0119828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0119951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0120127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0120399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0120519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0120794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0120927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0121101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0121401Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0121694Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0121843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0122003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0122076Z dist init r=1, world=2
2023-01-11T21:09:53.0122150Z dist init r=0, world=2
2023-01-11T21:09:53.0122250Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0122549Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28376
2023-01-11T21:09:53.0122702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28377
2023-01-11T21:09:53.0122973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0123085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0123363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0123494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0123669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0123938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0124058Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0124333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0124465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0124668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0124952Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0125245Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0125406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0125566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0125640Z dist init r=1, world=2
2023-01-11T21:09:53.0125710Z dist init r=0, world=2
2023-01-11T21:09:53.0125811Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0126128Z   test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28404
2023-01-11T21:09:53.0126274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28405
2023-01-11T21:09:53.0126549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0126669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0126947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0127079Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0127253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0127524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0127646Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0127925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0128045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0128218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0128513Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0128808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0128971Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0129128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0129201Z dist init r=1, world=2
2023-01-11T21:09:53.0129277Z dist init r=0, world=2
2023-01-11T21:09:53.0129367Z skip: Need at least 2 CUDA devices (1.615s)
2023-01-11T21:09:53.0129655Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28432
2023-01-11T21:09:53.0129811Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28433
2023-01-11T21:09:53.0130079Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0130200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0130479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0130611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0130790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0131253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0131360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0131639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0131772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0131947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0132245Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0132543Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0132757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0132920Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0132993Z dist init r=1, world=2
2023-01-11T21:09:53.0133052Z dist init r=0, world=2
2023-01-11T21:09:53.0133156Z skip: Need at least 2 CUDA devices (1.615s)
2023-01-11T21:09:53.0133440Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28460
2023-01-11T21:09:53.0133595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28461
2023-01-11T21:09:53.0133869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0133992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0134274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0134411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0134571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0134841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0134964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0135243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0135377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0135550Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0135847Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0136145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0136305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0136452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0136526Z dist init r=1, world=2
2023-01-11T21:09:53.0136596Z dist init r=0, world=2
2023-01-11T21:09:53.0136700Z skip: Need at least 2 CUDA devices (1.615s)
2023-01-11T21:09:53.0136984Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28488
2023-01-11T21:09:53.0137137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28489
2023-01-11T21:09:53.0137410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0137569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0137837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0137969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0138140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0138411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0138530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0138808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0138941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0139148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0139497Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0139781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0139943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0140105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0140178Z dist init r=1, world=2
2023-01-11T21:09:53.0140250Z dist init r=0, world=2
2023-01-11T21:09:53.0140353Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0140638Z   test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28516
2023-01-11T21:09:53.0140795Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28517
2023-01-11T21:09:53.0141054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0141177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0141460Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0141595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0141771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0142039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0142165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0142445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0142582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0142743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0143039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0143332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0143494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0143654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0143730Z dist init r=0, world=2
2023-01-11T21:09:53.0143839Z dist init r=1, world=2
2023-01-11T21:09:53.0143940Z skip: Need at least 2 CUDA devices (1.615s)
2023-01-11T21:09:53.0144134Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0144365Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28544
2023-01-11T21:09:53.0144519Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28545
2023-01-11T21:09:53.0144788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0144909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0145187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0145347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0145522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0145791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0145900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0146179Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0146309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0146483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0146779Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0147075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0147239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0147398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0147471Z dist init r=0, world=2
2023-01-11T21:09:53.0147529Z dist init r=1, world=2
2023-01-11T21:09:53.0147634Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0147841Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0148068Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28572
2023-01-11T21:09:53.0148221Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28573
2023-01-11T21:09:53.0148492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0148614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0148892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0149012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0149190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0149459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0149578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0149858Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0149988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0150196Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0150494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0150789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0150939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0151098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0151171Z dist init r=1, world=2
2023-01-11T21:09:53.0151242Z dist init r=0, world=2
2023-01-11T21:09:53.0151343Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0151540Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_False (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0151794Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28600
2023-01-11T21:09:53.0151950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28601
2023-01-11T21:09:53.0152210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0152330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0152607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0152739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0152912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0153180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0153303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0153587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0153707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0153879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0154175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0154468Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0154629Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0154789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0154867Z dist init r=0, world=2
2023-01-11T21:09:53.0154937Z dist init r=1, world=2
2023-01-11T21:09:53.0155037Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0155223Z   test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_True (__main__.TestFSDPStateDict)
2023-01-11T21:09:53.0155452Z Tests saving and loading a state dict for a model manually wrapped with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28628
2023-01-11T21:09:53.0155606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28629
2023-01-11T21:09:53.0155876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0155996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0156274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0156441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0156615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0156873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0156993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0157270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0157400Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0157574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0157871Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0158193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0158361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0158522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0158583Z dist init r=1, world=2
2023-01-11T21:09:53.0158658Z dist init r=0, world=2
2023-01-11T21:09:53.0158764Z skip: Need at least 2 CUDA devices (1.614s)
2023-01-11T21:09:53.0159029Z   test_state_dict_with_shared_parameters_state_dict_type_local_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28656
2023-01-11T21:09:53.0159184Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28657
2023-01-11T21:09:53.0159454Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0159579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0159862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0159982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0160156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0160428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0160552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0160831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0160964Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0161139Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0161441Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0161732Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0161883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0162042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0162117Z dist init r=0, world=2
2023-01-11T21:09:53.0162189Z dist init r=1, world=2
2023-01-11T21:09:53.0162294Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:09:53.0162561Z   test_state_dict_with_shared_parameters_state_dict_type_sharded_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28684
2023-01-11T21:09:53.0162720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28685
2023-01-11T21:09:53.0163062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0163171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0163451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0163585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0163760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0164028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0164149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0164456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0164592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0164766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0165050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0165343Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0165506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0165669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0165744Z dist init r=1, world=2
2023-01-11T21:09:53.0165819Z dist init r=0, world=2
2023-01-11T21:09:53.0165920Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:09:53.0166183Z   test_state_dict_with_shared_parameters_state_dict_type_state_dict (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28712
2023-01-11T21:09:53.0166328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28713
2023-01-11T21:09:53.0166600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0166723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0167002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0167136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0167310Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0167581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0167707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0167986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0168110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0168289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0168587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0168881Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0169040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0169199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0169303Z dist init r=0, world=2
2023-01-11T21:09:53.0169377Z dist init r=1, world=2
2023-01-11T21:09:53.0169467Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:09:53.0169688Z   test_wrong_state_dict_config (__main__.TestFSDPStateDict) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28740
2023-01-11T21:09:53.0169841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28741
2023-01-11T21:09:53.0170113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0170232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0170508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0170640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0171010Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:09:53.0171299Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:09:53.0171409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:09:53.0171689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:09:53.0171820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:09:53.0171994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:09:53.0172290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0172583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:09:53.0172751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:09:53.0172910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:09:53.0172972Z dist init r=0, world=2
2023-01-11T21:09:53.0173046Z dist init r=1, world=2
2023-01-11T21:09:53.0173148Z skip: Need at least 2 CUDA devices (1.613s)
2023-01-11T21:09:53.0173157Z 
2023-01-11T21:09:53.0173356Z ----------------------------------------------------------------------
2023-01-11T21:09:53.0173435Z Ran 116 tests in 187.592s
2023-01-11T21:09:53.0173440Z 
2023-01-11T21:09:53.0173515Z OK (skipped=116)
2023-01-11T21:09:53.0173520Z 
2023-01-11T21:09:53.0173602Z Generating XML reports...
2023-01-11T21:09:53.0173937Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20230111210644.xml
2023-01-11T21:09:53.0173942Z 
2023-01-11T21:09:53.0174414Z ##[endgroup]
2023-01-11T21:09:53.0174770Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_state_dict (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_state_dict_azd3szu2)
2023-01-11T21:09:53.0174777Z 
2023-01-11T21:09:53.0174974Z Running distributed/rpc/test_faulty_agent ... [2023-01-11 21:09:52.941331]
2023-01-11T21:09:53.0175362Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_faulty_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:09:52.941689]
2023-01-11T21:14:25.5393374Z 
2023-01-11T21:14:25.5394193Z Expand the folded group to see the log file of distributed/rpc/test_faulty_agent
2023-01-11T21:14:25.5395490Z ##[group]PRINTING LOG FILE of distributed/rpc/test_faulty_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_faulty_agent_32hjgg0g)
2023-01-11T21:14:25.5400539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8_ia2atu
2023-01-11T21:14:25.5401488Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8_ia2atu/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5402807Z <unittest.suite.TestSuite tests=[<__main__.FaultyFaultyAgentDistAutogradTest testMethod=test_context_cleanup_tensor_with_grad>, <__main__.FaultyFaultyAgentDistAutogradTest testMethod=test_verify_backend_options>]>
2023-01-11T21:14:25.5403755Z test_context_cleanup_tensor_with_grad (__main__.FaultyFaultyAgentDistAutogradTest)
2023-01-11T21:14:25.5404823Z test_verify_backend_options (__main__.FaultyFaultyAgentDistAutogradTest)
2023-01-11T21:14:25.5408360Z <unittest.suite.TestSuite tests=[<__main__.FaultyFaultyAgentRpcTest testMethod=test_builtin_remote_message_dropped_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_builtin_remote_message_dropped_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_check_failed_messages>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_custom_faulty_messages>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_custom_messages_to_delay>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_no_faulty_messages>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_builtin_delay_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_builtin_delay_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_dropped_pickle>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_dropped_pickle_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_script_delay_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_remote_message_script_delay_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_rpc_builtin_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_rpc_script_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_rref_to_here_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_delay_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_delay_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_dropped_timeout>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_udf_remote_message_dropped_timeout_to_self>, <__main__.FaultyFaultyAgentRpcTest testMethod=test_verify_backend_options>]>
2023-01-11T21:14:25.5411821Z test_builtin_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5412358Z test_builtin_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5412804Z test_check_failed_messages (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5413221Z test_custom_faulty_messages (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5413657Z test_custom_messages_to_delay (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5414088Z test_no_faulty_messages (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5451748Z test_remote_message_builtin_delay_timeout (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5452311Z test_remote_message_builtin_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5452936Z test_remote_message_dropped_pickle (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5453424Z test_remote_message_dropped_pickle_to_self (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5453784Z test_remote_message_script_delay_timeout (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5454120Z test_remote_message_script_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5454433Z test_rpc_builtin_timeout (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5454720Z test_rpc_script_timeout (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5454993Z test_rref_to_here_timeout (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5455294Z test_udf_remote_message_delay_timeout (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5455619Z test_udf_remote_message_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5455934Z test_udf_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5456395Z test_udf_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5456710Z test_verify_backend_options (__main__.FaultyFaultyAgentRpcTest)
2023-01-11T21:14:25.5457636Z <unittest.suite.TestSuite tests=[<__main__.FaultyJitFaultyAgentRpcTest testMethod=test_remote_timeout_to_here_in_jit>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_rref_timeout_pickle_in_jit>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_rref_timeout_pickle_script_func>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_rref_to_here_timeout_in_jit>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_timeout_in_python>, <__main__.FaultyJitFaultyAgentRpcTest testMethod=test_timeout_in_torchscript_function>]>
2023-01-11T21:14:25.5458475Z test_remote_timeout_to_here_in_jit (__main__.FaultyJitFaultyAgentRpcTest)
2023-01-11T21:14:25.5458791Z test_rref_timeout_pickle_in_jit (__main__.FaultyJitFaultyAgentRpcTest)
2023-01-11T21:14:25.5459158Z test_rref_timeout_pickle_script_func (__main__.FaultyJitFaultyAgentRpcTest)
2023-01-11T21:14:25.5459484Z test_rref_to_here_timeout_in_jit (__main__.FaultyJitFaultyAgentRpcTest)
2023-01-11T21:14:25.5459788Z test_timeout_in_python (__main__.FaultyJitFaultyAgentRpcTest)
2023-01-11T21:14:25.5460088Z test_timeout_in_torchscript_function (__main__.FaultyJitFaultyAgentRpcTest)
2023-01-11T21:14:25.5460731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5461066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5461498Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5461832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5462175Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgwj6rd3q
2023-01-11T21:14:25.5462577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgwj6rd3q/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5463039Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5463226Z 
2023-01-11T21:14:25.5463298Z Running tests...
2023-01-11T21:14:25.5465123Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5465896Z   test_context_cleanup_tensor_with_grad (__main__.FaultyFaultyAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28789
2023-01-11T21:14:25.5466635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28790
2023-01-11T21:14:25.5467113Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28791
2023-01-11T21:14:25.5467660Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28792
2023-01-11T21:14:25.5468452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5469020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5469819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5470380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5470993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbjp884ih
2023-01-11T21:14:25.5471731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbjp884ih/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5472623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5473162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5473825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5474171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5474603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptntdybc2
2023-01-11T21:14:25.5475112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptntdybc2/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5475962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5476342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5476968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5477523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5478112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphh82wmwh
2023-01-11T21:14:25.5478795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphh82wmwh/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5479700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5480262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5480987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5481547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5482110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkd49ry6p
2023-01-11T21:14:25.5482752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkd49ry6p/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5483364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5483919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5484489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5485033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5485837Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:14:25.5486509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:14:25.5487175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:14:25.5487826Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:14:25.5488754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:14:25.5489707Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:14:25.5490278Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:14:25.5490792Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:14:25.5491342Z ok (9.644s)
2023-01-11T21:14:25.5491453Z 
2023-01-11T21:14:25.5492003Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5492386Z Ran 1 test in 9.644s
2023-01-11T21:14:25.5492569Z 
2023-01-11T21:14:25.5492675Z OK
2023-01-11T21:14:25.5492826Z 
2023-01-11T21:14:25.5492948Z Generating XML reports...
2023-01-11T21:14:25.5493805Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20230111210955.xml
2023-01-11T21:14:25.5494768Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5495547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5495972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5496358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5496715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpakpj7l01
2023-01-11T21:14:25.5497120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpakpj7l01/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5497571Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5497772Z 
2023-01-11T21:14:25.5497846Z Running tests...
2023-01-11T21:14:25.5498152Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5498622Z   test_verify_backend_options (__main__.FaultyFaultyAgentDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28976
2023-01-11T21:14:25.5499053Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28977
2023-01-11T21:14:25.5499383Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 28978
2023-01-11T21:14:25.5499704Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 28979
2023-01-11T21:14:25.5500143Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5500533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5500961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5501306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5501635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7fxksbvp
2023-01-11T21:14:25.5502039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7fxksbvp/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5502525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5502837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5503257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5503596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5503935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq942mudi
2023-01-11T21:14:25.5504319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq942mudi/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5504799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5505123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5505547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5505871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5506212Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpovfrw4wm
2023-01-11T21:14:25.5506609Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpovfrw4wm/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5507076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5507399Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5507821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5508159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5508527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp80cgxinl
2023-01-11T21:14:25.5508925Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp80cgxinl/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5509295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5509630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5509969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5510304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5510552Z ok (8.340s)
2023-01-11T21:14:25.5510641Z 
2023-01-11T21:14:25.5510844Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5511080Z Ran 1 test in 8.340s
2023-01-11T21:14:25.5511195Z 
2023-01-11T21:14:25.5511257Z OK
2023-01-11T21:14:25.5511350Z 
2023-01-11T21:14:25.5511452Z Generating XML reports...
2023-01-11T21:14:25.5511960Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20230111211007.xml
2023-01-11T21:14:25.5512533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5512862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5513276Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5513621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5513963Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph23b026n
2023-01-11T21:14:25.5514357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph23b026n/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5514796Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5514997Z 
2023-01-11T21:14:25.5515071Z Running tests...
2023-01-11T21:14:25.5515373Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5515778Z   test_builtin_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29151
2023-01-11T21:14:25.5516191Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29152
2023-01-11T21:14:25.5516516Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29153
2023-01-11T21:14:25.5516838Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29154
2023-01-11T21:14:25.5517274Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5517604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5518033Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5518362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5518699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8g14dsjw
2023-01-11T21:14:25.5519094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8g14dsjw/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5519577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5519889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5520311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5520652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5520991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnkx8ermx
2023-01-11T21:14:25.5521410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnkx8ermx/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5521898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5522219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5522633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5522971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5523310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfoo49esd
2023-01-11T21:14:25.5523705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfoo49esd/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5524208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5524537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5524958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5525282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5525619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp4jhdsun
2023-01-11T21:14:25.5526016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp4jhdsun/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5526391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5526723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5527054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5527398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5527644Z ok (8.343s)
2023-01-11T21:14:25.5527733Z 
2023-01-11T21:14:25.5527936Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5528173Z Ran 1 test in 8.343s
2023-01-11T21:14:25.5528282Z 
2023-01-11T21:14:25.5528342Z OK
2023-01-11T21:14:25.5528421Z 
2023-01-11T21:14:25.5528505Z Generating XML reports...
2023-01-11T21:14:25.5528971Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211017.xml
2023-01-11T21:14:25.5529514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5529840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5530254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5530603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5531142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpggu07kq1
2023-01-11T21:14:25.5531532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpggu07kq1/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5531995Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5532194Z 
2023-01-11T21:14:25.5532266Z Running tests...
2023-01-11T21:14:25.5532569Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5532978Z   test_builtin_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29326
2023-01-11T21:14:25.5533398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29327
2023-01-11T21:14:25.5533730Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29328
2023-01-11T21:14:25.5534132Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29329
2023-01-11T21:14:25.5534589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5534917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5535343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5535675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5536013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgabgvb7g
2023-01-11T21:14:25.5536413Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgabgvb7g/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5536941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5537258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5537787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5538131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5538462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6miub5_i
2023-01-11T21:14:25.5538858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6miub5_i/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5539341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5539668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5540078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5540497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5540836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2npcxu__
2023-01-11T21:14:25.5541221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2npcxu__/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5541703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5542024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5542445Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5542773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5543110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd6lwzdyt
2023-01-11T21:14:25.5543509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd6lwzdyt/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5543886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5544221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5544558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5544900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5545135Z ok (8.343s)
2023-01-11T21:14:25.5545237Z 
2023-01-11T21:14:25.5545438Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5545675Z Ran 1 test in 8.343s
2023-01-11T21:14:25.5545787Z 
2023-01-11T21:14:25.5545848Z OK
2023-01-11T21:14:25.5545925Z 
2023-01-11T21:14:25.5546009Z Generating XML reports...
2023-01-11T21:14:25.5546479Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211027.xml
2023-01-11T21:14:25.5547066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5547380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5547807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5548149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5548489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo_tcw2sp
2023-01-11T21:14:25.5548877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo_tcw2sp/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5549331Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5549530Z 
2023-01-11T21:14:25.5549603Z Running tests...
2023-01-11T21:14:25.5549923Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5550329Z   test_check_failed_messages (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29501
2023-01-11T21:14:25.5550727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29502
2023-01-11T21:14:25.5551053Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29503
2023-01-11T21:14:25.5551360Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29504
2023-01-11T21:14:25.5551803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5552126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5552550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5552885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5553231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcpmhlkw5
2023-01-11T21:14:25.5553632Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcpmhlkw5/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5554099Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5554422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5554843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5555181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5555509Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx892gc2y
2023-01-11T21:14:25.5555905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx892gc2y/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5556387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5556698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5557119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5557457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5557796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6wv0hjc
2023-01-11T21:14:25.5558178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6wv0hjc/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5558655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5558977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5559399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5559758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5560096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl7ubd6rz
2023-01-11T21:14:25.5560492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl7ubd6rz/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5560852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5561195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5561536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5561874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5562109Z ok (13.050s)
2023-01-11T21:14:25.5562210Z 
2023-01-11T21:14:25.5562440Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5562681Z Ran 1 test in 13.050s
2023-01-11T21:14:25.5562790Z 
2023-01-11T21:14:25.5562837Z OK
2023-01-11T21:14:25.5562928Z 
2023-01-11T21:14:25.5563016Z Generating XML reports...
2023-01-11T21:14:25.5563480Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211037.xml
2023-01-11T21:14:25.5564021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5564333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5564755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5565097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5565424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp681qrdnb
2023-01-11T21:14:25.5565825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp681qrdnb/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5566281Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5566479Z 
2023-01-11T21:14:25.5566549Z Running tests...
2023-01-11T21:14:25.5566836Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5567233Z   test_custom_faulty_messages (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29676
2023-01-11T21:14:25.5567633Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29677
2023-01-11T21:14:25.5567946Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29678
2023-01-11T21:14:25.5568266Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29679
2023-01-11T21:14:25.5568711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5569044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5569456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5569800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5570140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58gisjg0
2023-01-11T21:14:25.5570536Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58gisjg0/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5571174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5571503Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5571932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5572330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5572671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppkrmoxlc
2023-01-11T21:14:25.5573072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppkrmoxlc/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5573554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5573864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5574285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5574625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5574949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1488x7ja
2023-01-11T21:14:25.5575383Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1488x7ja/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5575964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5576292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5576699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5577037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5577375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpptovvs6x
2023-01-11T21:14:25.5577772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpptovvs6x/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5578130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5578475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5578817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5579146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5579395Z ok (8.348s)
2023-01-11T21:14:25.5579494Z 
2023-01-11T21:14:25.5579693Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5579932Z Ran 1 test in 8.348s
2023-01-11T21:14:25.5580030Z 
2023-01-11T21:14:25.5580090Z OK
2023-01-11T21:14:25.5580180Z 
2023-01-11T21:14:25.5580328Z Generating XML reports...
2023-01-11T21:14:25.5580810Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211052.xml
2023-01-11T21:14:25.5581335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5581663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5582088Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5582431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5582760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkex7oain
2023-01-11T21:14:25.5583156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkex7oain/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5583609Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5583806Z 
2023-01-11T21:14:25.5583878Z Running tests...
2023-01-11T21:14:25.5584166Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5584566Z   test_custom_messages_to_delay (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29851
2023-01-11T21:14:25.5585013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29852
2023-01-11T21:14:25.5585327Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 29853
2023-01-11T21:14:25.5585647Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 29854
2023-01-11T21:14:25.5586096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5586576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5587004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5587344Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5587685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn4aqp1ak
2023-01-11T21:14:25.5588108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn4aqp1ak/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5588596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5588924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5589344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5589669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5590008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0rg99quk
2023-01-11T21:14:25.5590402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0rg99quk/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5590873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5591203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5591635Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5591979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5592305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mh1r8u8
2023-01-11T21:14:25.5592701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mh1r8u8/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5593182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5593504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5593913Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5594253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5594596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv_90tf2f
2023-01-11T21:14:25.5594979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv_90tf2f/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5595350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5595694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5596036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5596364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5596611Z ok (2.531s)
2023-01-11T21:14:25.5596712Z 
2023-01-11T21:14:25.5596917Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5597210Z Ran 1 test in 2.532s
2023-01-11T21:14:25.5597373Z 
2023-01-11T21:14:25.5597444Z OK
2023-01-11T21:14:25.5597536Z 
2023-01-11T21:14:25.5597679Z Generating XML reports...
2023-01-11T21:14:25.5598155Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211103.xml
2023-01-11T21:14:25.5598689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5599015Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5599438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5599770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5600108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjdvk674b
2023-01-11T21:14:25.5600507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjdvk674b/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5601492Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5601698Z 
2023-01-11T21:14:25.5601757Z Running tests...
2023-01-11T21:14:25.5602058Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5602454Z   test_no_faulty_messages (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30026
2023-01-11T21:14:25.5602848Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30027
2023-01-11T21:14:25.5603161Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30028
2023-01-11T21:14:25.5603481Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30029
2023-01-11T21:14:25.5603928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5604241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5604668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5605013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5605360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa8jzlp_t
2023-01-11T21:14:25.5605751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa8jzlp_t/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5606233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5606555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5606979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5607311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5607651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx8utpqq8
2023-01-11T21:14:25.5608052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx8utpqq8/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5608520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5608841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5609261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5609598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5609924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoizlpmcp
2023-01-11T21:14:25.5610323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoizlpmcp/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5610804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5611294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5611726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5612067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5612405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw7zc7vwf
2023-01-11T21:14:25.5612788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw7zc7vwf/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5613161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5613503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5613843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5614222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5614469Z ok (8.341s)
2023-01-11T21:14:25.5614571Z 
2023-01-11T21:14:25.5614773Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5614997Z Ran 1 test in 8.341s
2023-01-11T21:14:25.5615107Z 
2023-01-11T21:14:25.5615167Z OK
2023-01-11T21:14:25.5615259Z 
2023-01-11T21:14:25.5615342Z Generating XML reports...
2023-01-11T21:14:25.5615798Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211107.xml
2023-01-11T21:14:25.5616341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5616664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5617090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5617421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5617765Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw560kxbp
2023-01-11T21:14:25.5618164Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw560kxbp/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5618620Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5618807Z 
2023-01-11T21:14:25.5618878Z Running tests...
2023-01-11T21:14:25.5619178Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5619592Z   test_remote_message_builtin_delay_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30201
2023-01-11T21:14:25.5636253Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30202
2023-01-11T21:14:25.5636687Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30203
2023-01-11T21:14:25.5637031Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30204
2023-01-11T21:14:25.5637531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5637853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5638283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5638629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5638976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp16rygzyr
2023-01-11T21:14:25.5639362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp16rygzyr/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5639847Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5640176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5640742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5641085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5641429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpemcg8gqm
2023-01-11T21:14:25.5641831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpemcg8gqm/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5642303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5642628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5643053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5643393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5643763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnutgmnrp
2023-01-11T21:14:25.5644163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnutgmnrp/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5644647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5644962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5645386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5645745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5646088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz5tlavno
2023-01-11T21:14:25.5646471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz5tlavno/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5646848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5647192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5647536Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5647865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5648108Z ok (7.447s)
2023-01-11T21:14:25.5648210Z 
2023-01-11T21:14:25.5648412Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5648640Z Ran 1 test in 7.447s
2023-01-11T21:14:25.5648750Z 
2023-01-11T21:14:25.5648809Z OK
2023-01-11T21:14:25.5648901Z 
2023-01-11T21:14:25.5648984Z Generating XML reports...
2023-01-11T21:14:25.5649442Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211117.xml
2023-01-11T21:14:25.5649990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5650319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5650743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5651230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5651573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpltv8yita
2023-01-11T21:14:25.5651974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpltv8yita/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5652435Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5652623Z 
2023-01-11T21:14:25.5652694Z Running tests...
2023-01-11T21:14:25.5652994Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5653477Z   test_remote_message_builtin_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30376
2023-01-11T21:14:25.5653885Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30377
2023-01-11T21:14:25.5654214Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30378
2023-01-11T21:14:25.5654537Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30379
2023-01-11T21:14:25.5654982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5655293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5655716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5656058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5656424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvydwzjif
2023-01-11T21:14:25.5656827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvydwzjif/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5657312Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5657636Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5658046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5658382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5658718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplxa4cwee
2023-01-11T21:14:25.5659111Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplxa4cwee/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5659580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5659908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5660393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5660722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5661063Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbrl89o9q
2023-01-11T21:14:25.5661453Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbrl89o9q/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5661938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5662249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5662673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5663014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5663352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgm1eg1o3
2023-01-11T21:14:25.5663733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgm1eg1o3/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5664106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5664448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5664779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5665114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5665683Z [E request_callback_no_python.cpp:559] Received error while processing request type 260: false INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp":395, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created.
2023-01-11T21:14:25.5666381Z Exception raised from getOwnerRRef at /var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp:395 (most recent call first):
2023-01-11T21:14:25.5667054Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f40797ea3db in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2023-01-11T21:14:25.5667791Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xbf (0x7f40797e521f in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2023-01-11T21:14:25.5668610Z frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x4e (0x7f40797e5e1e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2023-01-11T21:14:25.5669350Z frame #3: torch::distributed::rpc::RRefContext::getOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, bool) + 0x450 (0x7f407e204020 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5670320Z frame #4: torch::distributed::rpc::RequestCallbackNoPython::assignOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&, c10::intrusive_ptr<c10::ivalue::Future, c10::detail::intrusive_target_default_null_type<c10::ivalue::Future> >) const + 0x5e4 (0x7f407e1f0c14 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5671294Z frame #5: torch::distributed::rpc::RequestCallbackImpl::processScriptRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x12a (0x7f408661986a in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
2023-01-11T21:14:25.5672221Z frame #6: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x14c (0x7f407e1f7c8c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5673178Z frame #7: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f4086614ba5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
2023-01-11T21:14:25.5673841Z frame #8: <unknown function> + 0x49b9433 (0x7f407e1ef433 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5674533Z frame #9: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x535 (0x7f407e1f3525 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5675344Z frame #10: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f407e1eb047 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5675927Z frame #11: <unknown function> + 0x49e9990 (0x7f407e21f990 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5676428Z frame #12: c10::ThreadPool::main_loop(unsigned long) + 0x2eb (0x7f40797d7efb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2023-01-11T21:14:25.5676779Z frame #13: <unknown function> + 0xdbbf4 (0x7f4079659bf4 in /opt/conda/lib/libstdc++.so.6)
2023-01-11T21:14:25.5677181Z frame #14: <unknown function> + 0x8609 (0x7f408e61e609 in /lib/x86_64-linux-gnu/libpthread.so.0)
2023-01-11T21:14:25.5677584Z frame #15: clone + 0x43 (0x7f408e3e9133 in /lib/x86_64-linux-gnu/libc.so.6)
2023-01-11T21:14:25.5677739Z 
2023-01-11T21:14:25.5677806Z ok (4.332s)
2023-01-11T21:14:25.5677906Z 
2023-01-11T21:14:25.5678094Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5678332Z Ran 1 test in 4.333s
2023-01-11T21:14:25.5678441Z 
2023-01-11T21:14:25.5678502Z OK
2023-01-11T21:14:25.5678591Z 
2023-01-11T21:14:25.5678673Z Generating XML reports...
2023-01-11T21:14:25.5679125Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211126.xml
2023-01-11T21:14:25.5679667Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5679992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5680432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5680782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5681121Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7b_yzl58
2023-01-11T21:14:25.5681516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7b_yzl58/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5681955Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5682155Z 
2023-01-11T21:14:25.5682227Z Running tests...
2023-01-11T21:14:25.5682526Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5682922Z   test_remote_message_dropped_pickle (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30551
2023-01-11T21:14:25.5683328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30552
2023-01-11T21:14:25.5683663Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30553
2023-01-11T21:14:25.5683987Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30554
2023-01-11T21:14:25.5684421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5684745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5685166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5685507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5685835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr0rimrzi
2023-01-11T21:14:25.5686231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr0rimrzi/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5686715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5687029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5687447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5687789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5688122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzsk0bqki
2023-01-11T21:14:25.5688509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzsk0bqki/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5688993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5689315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5689737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5690094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5690431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7uf_7y1b
2023-01-11T21:14:25.5690954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7uf_7y1b/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5691455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5691767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5692193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5692534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5692862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjxag2gts
2023-01-11T21:14:25.5693320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjxag2gts/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5693698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5694045Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5694376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5694713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5694967Z ok (8.333s)
2023-01-11T21:14:25.5695071Z 
2023-01-11T21:14:25.5695258Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5695496Z Ran 1 test in 8.333s
2023-01-11T21:14:25.5695606Z 
2023-01-11T21:14:25.5695666Z OK
2023-01-11T21:14:25.5695756Z 
2023-01-11T21:14:25.5695839Z Generating XML reports...
2023-01-11T21:14:25.5696298Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211133.xml
2023-01-11T21:14:25.5696843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5697171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5697580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5697917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5698258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmporzjt3uo
2023-01-11T21:14:25.5698656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmporzjt3uo/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5699098Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5699294Z 
2023-01-11T21:14:25.5699365Z Running tests...
2023-01-11T21:14:25.5699671Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5700072Z   test_remote_message_dropped_pickle_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30726
2023-01-11T21:14:25.5700554Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30727
2023-01-11T21:14:25.5700883Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30728
2023-01-11T21:14:25.5701211Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30729
2023-01-11T21:14:25.5701647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5701974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5702399Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5702796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5703124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkoc9k2cb
2023-01-11T21:14:25.5703519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkoc9k2cb/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5703997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5704311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5704733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5705072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5705409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdrl_2ot7
2023-01-11T21:14:25.5705820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdrl_2ot7/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5706307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5706630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5707050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5707375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5707712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoaiqjcz5
2023-01-11T21:14:25.5708107Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoaiqjcz5/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5708574Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5708891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5709318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5709659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5709984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkd2qe4ed
2023-01-11T21:14:25.5710376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkd2qe4ed/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5710747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5711075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5711414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5711748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5711993Z ok (8.339s)
2023-01-11T21:14:25.5712086Z 
2023-01-11T21:14:25.5712281Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5712517Z Ran 1 test in 8.340s
2023-01-11T21:14:25.5712626Z 
2023-01-11T21:14:25.5712685Z OK
2023-01-11T21:14:25.5712774Z 
2023-01-11T21:14:25.5712846Z Generating XML reports...
2023-01-11T21:14:25.5713313Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211143.xml
2023-01-11T21:14:25.5713855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5714187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5714604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5714949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5715296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4mg4ac08
2023-01-11T21:14:25.5715730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4mg4ac08/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5716185Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5716384Z 
2023-01-11T21:14:25.5716458Z Running tests...
2023-01-11T21:14:25.5716762Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5717165Z   test_remote_message_script_delay_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30901
2023-01-11T21:14:25.5717583Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30902
2023-01-11T21:14:25.5717913Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 30903
2023-01-11T21:14:25.5718241Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 30904
2023-01-11T21:14:25.5718705Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5719033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5719465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5719793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5720135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_32svyq6
2023-01-11T21:14:25.5720531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_32svyq6/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5721016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5721330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5721757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5722104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5722432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplb5d1p25
2023-01-11T21:14:25.5722828Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplb5d1p25/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5723317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5723640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5724052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5724393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5724737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp19f4pqd0
2023-01-11T21:14:25.5725132Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp19f4pqd0/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5725600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5725925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5726347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5726672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5727015Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_r1rwvc_
2023-01-11T21:14:25.5727407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_r1rwvc_/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5727776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5728138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5728476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5728819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5729054Z ok (7.438s)
2023-01-11T21:14:25.5729152Z 
2023-01-11T21:14:25.5729352Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5729587Z Ran 1 test in 7.438s
2023-01-11T21:14:25.5729696Z 
2023-01-11T21:14:25.5729758Z OK
2023-01-11T21:14:25.5729835Z 
2023-01-11T21:14:25.5729920Z Generating XML reports...
2023-01-11T21:14:25.5730387Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211153.xml
2023-01-11T21:14:25.5731075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5731450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5731881Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5732224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5732567Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1vkvi41q
2023-01-11T21:14:25.5732959Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1vkvi41q/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5733415Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5733610Z 
2023-01-11T21:14:25.5733684Z Running tests...
2023-01-11T21:14:25.5733982Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5734394Z   test_remote_message_script_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31076
2023-01-11T21:14:25.5734819Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31077
2023-01-11T21:14:25.5735147Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31078
2023-01-11T21:14:25.5735461Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31079
2023-01-11T21:14:25.5735906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5736234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5736663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5736993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5737335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0o0pbtfm
2023-01-11T21:14:25.5737735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0o0pbtfm/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5738215Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5738526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5738948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5739287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5739611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsvbbvqt1
2023-01-11T21:14:25.5740008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsvbbvqt1/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5740565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5740942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5741354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5741690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5742027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5o0428gz
2023-01-11T21:14:25.5742405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5o0428gz/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5742882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5743206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5743631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5743989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5744330Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0a3s9i5m
2023-01-11T21:14:25.5744726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0a3s9i5m/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5745098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5745427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5745762Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5746103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5746655Z [E request_callback_no_python.cpp:559] Received error while processing request type 260: false INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp":395, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created.
2023-01-11T21:14:25.5747256Z Exception raised from getOwnerRRef at /var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp:395 (most recent call first):
2023-01-11T21:14:25.5747903Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fddd38d33db in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2023-01-11T21:14:25.5748642Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xbf (0x7fddd38ce21f in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2023-01-11T21:14:25.5749451Z frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x4e (0x7fddd38cee1e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2023-01-11T21:14:25.5750200Z frame #3: torch::distributed::rpc::RRefContext::getOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, bool) + 0x450 (0x7fddd82ed020 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5751158Z frame #4: torch::distributed::rpc::RequestCallbackNoPython::assignOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&, c10::intrusive_ptr<c10::ivalue::Future, c10::detail::intrusive_target_default_null_type<c10::ivalue::Future> >) const + 0x5e4 (0x7fddd82d9c14 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5752133Z frame #5: torch::distributed::rpc::RequestCallbackImpl::processScriptRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x12a (0x7fdde070286a in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
2023-01-11T21:14:25.5753100Z frame #6: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x14c (0x7fddd82e0c8c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5754070Z frame #7: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7fdde06fdba5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
2023-01-11T21:14:25.5754736Z frame #8: <unknown function> + 0x49b9433 (0x7fddd82d8433 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5755469Z frame #9: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x535 (0x7fddd82dc525 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5756282Z frame #10: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7fddd82d4047 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5756853Z frame #11: <unknown function> + 0x49e9990 (0x7fddd8308990 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5757360Z frame #12: c10::ThreadPool::main_loop(unsigned long) + 0x2eb (0x7fddd38c0efb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2023-01-11T21:14:25.5757724Z frame #13: <unknown function> + 0xdbbf4 (0x7fddd3742bf4 in /opt/conda/lib/libstdc++.so.6)
2023-01-11T21:14:25.5758128Z frame #14: <unknown function> + 0x8609 (0x7fdde8707609 in /lib/x86_64-linux-gnu/libpthread.so.0)
2023-01-11T21:14:25.5758485Z frame #15: clone + 0x43 (0x7fdde84d2133 in /lib/x86_64-linux-gnu/libc.so.6)
2023-01-11T21:14:25.5758644Z 
2023-01-11T21:14:25.5758710Z ok (4.327s)
2023-01-11T21:14:25.5758810Z 
2023-01-11T21:14:25.5759008Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5759232Z Ran 1 test in 4.327s
2023-01-11T21:14:25.5759340Z 
2023-01-11T21:14:25.5759400Z OK
2023-01-11T21:14:25.5759489Z 
2023-01-11T21:14:25.5759572Z Generating XML reports...
2023-01-11T21:14:25.5760039Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211202.xml
2023-01-11T21:14:25.5760572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5760898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5761321Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5761654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5761993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9fn88moy
2023-01-11T21:14:25.5762390Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9fn88moy/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5762845Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5763042Z 
2023-01-11T21:14:25.5763102Z Running tests...
2023-01-11T21:14:25.5763399Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5763794Z   test_rpc_builtin_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31251
2023-01-11T21:14:25.5764188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31252
2023-01-11T21:14:25.5764500Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31253
2023-01-11T21:14:25.5764854Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31254
2023-01-11T21:14:25.5765302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5765614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5766037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5766381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5766724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt0z3qz_2
2023-01-11T21:14:25.5767113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt0z3qz_2/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5767593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5767952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5768367Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5768703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5769043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnamqzsx
2023-01-11T21:14:25.5769440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnamqzsx/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5769910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5770231Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5770650Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5771157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5771485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcwtf6zkz
2023-01-11T21:14:25.5771885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcwtf6zkz/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5772371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5772681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5773101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5773444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5773785Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdf9kbmgx
2023-01-11T21:14:25.5774171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdf9kbmgx/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5774549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5774896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5775229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5775575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5775819Z ok (15.847s)
2023-01-11T21:14:25.5775924Z 
2023-01-11T21:14:25.5776124Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5776352Z Ran 1 test in 15.847s
2023-01-11T21:14:25.5776464Z 
2023-01-11T21:14:25.5776527Z OK
2023-01-11T21:14:25.5776619Z 
2023-01-11T21:14:25.5776704Z Generating XML reports...
2023-01-11T21:14:25.5777163Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211208.xml
2023-01-11T21:14:25.5777766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5778097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5778530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5778863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5779207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfbuk1g2f
2023-01-11T21:14:25.5779610Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfbuk1g2f/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5780071Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5780326Z 
2023-01-11T21:14:25.5780401Z Running tests...
2023-01-11T21:14:25.5780709Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5781151Z   test_rpc_script_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31426
2023-01-11T21:14:25.5781540Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31427
2023-01-11T21:14:25.5781871Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31428
2023-01-11T21:14:25.5782200Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31429
2023-01-11T21:14:25.5782652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5782967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5783394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5783739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5784075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv6jjsy46
2023-01-11T21:14:25.5784482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv6jjsy46/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5784963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5785286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5785700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5786044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5786387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjhcbqqg8
2023-01-11T21:14:25.5786785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjhcbqqg8/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5787255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5787578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5787997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5788320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5788655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsw41kknx
2023-01-11T21:14:25.5789050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsw41kknx/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5789530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5789839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5790263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5790634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5790960Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpygt3xo0t
2023-01-11T21:14:25.5791355Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpygt3xo0t/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5791725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5792068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5792394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5792729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5792975Z ok (9.837s)
2023-01-11T21:14:25.5793076Z 
2023-01-11T21:14:25.5793272Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5793528Z Ran 1 test in 9.837s
2023-01-11T21:14:25.5793642Z 
2023-01-11T21:14:25.5793702Z OK
2023-01-11T21:14:25.5793791Z 
2023-01-11T21:14:25.5793875Z Generating XML reports...
2023-01-11T21:14:25.5794331Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211226.xml
2023-01-11T21:14:25.5794871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5795193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5795617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5795947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5796285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn8kym1o_
2023-01-11T21:14:25.5796684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn8kym1o_/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5797124Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5797322Z 
2023-01-11T21:14:25.5797394Z Running tests...
2023-01-11T21:14:25.5797698Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5798088Z   test_rref_to_here_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31601
2023-01-11T21:14:25.5798471Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31602
2023-01-11T21:14:25.5798797Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31603
2023-01-11T21:14:25.5799119Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31604
2023-01-11T21:14:25.5799552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5799881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5800303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5800642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5800971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaw2apuoq
2023-01-11T21:14:25.5801367Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaw2apuoq/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5801849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5802169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5802578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5802974Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5803311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptg_q_yrn
2023-01-11T21:14:25.5803696Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptg_q_yrn/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5804176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5804501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5804925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5805249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5805593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe1fkump0
2023-01-11T21:14:25.5806024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe1fkump0/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5806497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5806819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5807238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5807575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5807897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ot_4wyc
2023-01-11T21:14:25.5808287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ot_4wyc/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5808685Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5809028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5809359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5809696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5809942Z ok (4.529s)
2023-01-11T21:14:25.5810040Z 
2023-01-11T21:14:25.5810223Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5810459Z Ran 1 test in 4.529s
2023-01-11T21:14:25.5810569Z 
2023-01-11T21:14:25.5810630Z OK
2023-01-11T21:14:25.5810719Z 
2023-01-11T21:14:25.5810790Z Generating XML reports...
2023-01-11T21:14:25.5811447Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211238.xml
2023-01-11T21:14:25.5811989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5812319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5812738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5813081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5813422Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqzwwu_ll
2023-01-11T21:14:25.5813819Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqzwwu_ll/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5814259Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5814457Z 
2023-01-11T21:14:25.5814530Z Running tests...
2023-01-11T21:14:25.5814827Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5815222Z   test_udf_remote_message_delay_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31776
2023-01-11T21:14:25.5815630Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31777
2023-01-11T21:14:25.5816021Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31778
2023-01-11T21:14:25.5816349Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31779
2023-01-11T21:14:25.5816785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5817112Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5817537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5817866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5818203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_k5_f07
2023-01-11T21:14:25.5818677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5819044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5819457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5819798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5820135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmx2sucuy
2023-01-11T21:14:25.5820614Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_k5_f07/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5821002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmx2sucuy/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5821491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5821815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5822230Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5822574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5822912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy9vyodwp
2023-01-11T21:14:25.5823308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy9vyodwp/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5823777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5824098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5824519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5824860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5825192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyqyp3rh4
2023-01-11T21:14:25.5825589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyqyp3rh4/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5825958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5826285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5826620Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5826958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5827205Z ok (8.439s)
2023-01-11T21:14:25.5827293Z 
2023-01-11T21:14:25.5827495Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5827729Z Ran 1 test in 8.439s
2023-01-11T21:14:25.5827837Z 
2023-01-11T21:14:25.5827897Z OK
2023-01-11T21:14:25.5827988Z 
2023-01-11T21:14:25.5828058Z Generating XML reports...
2023-01-11T21:14:25.5828579Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211244.xml
2023-01-11T21:14:25.5829122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5829449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5829862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5830207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5830552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpga5xbe3e
2023-01-11T21:14:25.5830941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpga5xbe3e/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5831402Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5831605Z 
2023-01-11T21:14:25.5831708Z Running tests...
2023-01-11T21:14:25.5832016Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5832418Z   test_udf_remote_message_delay_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31951
2023-01-11T21:14:25.5832836Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31952
2023-01-11T21:14:25.5833161Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 31953
2023-01-11T21:14:25.5833472Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 31954
2023-01-11T21:14:25.5833924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5834249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5834682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5835016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5835361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdb9oj63j
2023-01-11T21:14:25.5835760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdb9oj63j/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5836245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5836556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5836983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5837326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5837654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqzzww4yx
2023-01-11T21:14:25.5838062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqzzww4yx/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5838547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5838874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5839284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5839620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5839959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprnqr7fj7
2023-01-11T21:14:25.5840344Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprnqr7fj7/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5840821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5841173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5841593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5841918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5842251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpks5nkdzt
2023-01-11T21:14:25.5842644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpks5nkdzt/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5843023Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5843354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5843687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5844029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5844612Z [E request_callback_no_python.cpp:559] Received error while processing request type 261: false INTERNAL ASSERT FAILED at "/var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp":395, please report a bug to PyTorch. Expected OwnerRRef with id GloballyUniqueId(created_on=0, local_id=0) to be created.
2023-01-11T21:14:25.5845209Z Exception raised from getOwnerRRef at /var/lib/jenkins/workspace/torch/csrc/distributed/rpc/rref_context.cpp:395 (most recent call first):
2023-01-11T21:14:25.5845859Z frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f4a2008d3db in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2023-01-11T21:14:25.5846598Z frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0xbf (0x7f4a2008821f in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2023-01-11T21:14:25.5847397Z frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x4e (0x7f4a20088e1e in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2023-01-11T21:14:25.5848139Z frame #3: torch::distributed::rpc::RRefContext::getOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, bool) + 0x450 (0x7f4a24aa7020 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5849100Z frame #4: torch::distributed::rpc::RequestCallbackNoPython::assignOwnerRRef(torch::distributed::rpc::GloballyUniqueId const&, torch::distributed::rpc::GloballyUniqueId const&, c10::intrusive_ptr<c10::ivalue::Future, c10::detail::intrusive_target_default_null_type<c10::ivalue::Future> >) const + 0x5e4 (0x7f4a24a93c14 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5850078Z frame #5: torch::distributed::rpc::RequestCallbackImpl::processPythonRemoteCall(torch::distributed::rpc::RpcCommandBase&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0xcc (0x7f4a2ceb849c in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
2023-01-11T21:14:25.5851197Z frame #6: torch::distributed::rpc::RequestCallbackNoPython::processRpc(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x194 (0x7f4a24a9acd4 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5852168Z frame #7: torch::distributed::rpc::RequestCallbackImpl::processRpcWithErrors(torch::distributed::rpc::RpcCommandBase&, torch::distributed::rpc::MessageType const&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x65 (0x7f4a2ceb7ba5 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_python.so)
2023-01-11T21:14:25.5852843Z frame #8: <unknown function> + 0x49b9433 (0x7f4a24a92433 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5853599Z frame #9: torch::distributed::rpc::RequestCallbackNoPython::processMessage(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x535 (0x7f4a24a96525 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5854869Z frame #10: torch::distributed::rpc::RequestCallback::operator()(torch::distributed::rpc::Message&, std::vector<c10::Stream, std::allocator<c10::Stream> >) const + 0x57 (0x7f4a24a8e047 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5855523Z frame #11: <unknown function> + 0x49e9990 (0x7f4a24ac2990 in /opt/conda/lib/python3.7/site-packages/torch/lib/libtorch_cpu.so)
2023-01-11T21:14:25.5856033Z frame #12: c10::ThreadPool::main_loop(unsigned long) + 0x2eb (0x7f4a2007aefb in /opt/conda/lib/python3.7/site-packages/torch/lib/libc10.so)
2023-01-11T21:14:25.5856469Z frame #13: <unknown function> + 0xdbbf4 (0x7f4a1fefcbf4 in /opt/conda/lib/libstdc++.so.6)
2023-01-11T21:14:25.5856877Z frame #14: <unknown function> + 0x8609 (0x7f4a34ec1609 in /lib/x86_64-linux-gnu/libpthread.so.0)
2023-01-11T21:14:25.5857229Z frame #15: clone + 0x43 (0x7f4a34c8c133 in /lib/x86_64-linux-gnu/libc.so.6)
2023-01-11T21:14:25.5857385Z 
2023-01-11T21:14:25.5857667Z [W tensorpipe_agent.cpp:624] RPC agent for worker0 won't send response to request #0 to worker0, as the agent is shutting down
2023-01-11T21:14:25.5857940Z ok (5.227s)
2023-01-11T21:14:25.5858042Z 
2023-01-11T21:14:25.5858241Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5858465Z Ran 1 test in 5.227s
2023-01-11T21:14:25.5858576Z 
2023-01-11T21:14:25.5858637Z OK
2023-01-11T21:14:25.5858728Z 
2023-01-11T21:14:25.5858814Z Generating XML reports...
2023-01-11T21:14:25.5859274Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211254.xml
2023-01-11T21:14:25.5859823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5860149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5860648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5860980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5861323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_guq5s9
2023-01-11T21:14:25.5861720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_guq5s9/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5862338Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5862659Z 
2023-01-11T21:14:25.5862736Z Running tests...
2023-01-11T21:14:25.5863041Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5863456Z   test_udf_remote_message_dropped_timeout (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32126
2023-01-11T21:14:25.5863856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32127
2023-01-11T21:14:25.5864182Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32128
2023-01-11T21:14:25.5864505Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32129
2023-01-11T21:14:25.5864955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5865269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5865691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5866031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5866415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph9l7rc3x
2023-01-11T21:14:25.5866891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5867212Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5867640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5867968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5868310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvb9ds7rk
2023-01-11T21:14:25.5868715Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph9l7rc3x/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5869104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvb9ds7rk/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5869620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5869941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5870362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5870686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5871026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30duwghu
2023-01-11T21:14:25.5871422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30duwghu/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5871907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5872219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5872647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5872782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5872964Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgzjqkjc
2023-01-11T21:14:25.5873156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgzjqkjc/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5873305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5873466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5873622Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5873780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5873848Z ok (8.335s)
2023-01-11T21:14:25.5873854Z 
2023-01-11T21:14:25.5874053Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5874130Z Ran 1 test in 8.335s
2023-01-11T21:14:25.5874135Z 
2023-01-11T21:14:25.5874193Z OK
2023-01-11T21:14:25.5874198Z 
2023-01-11T21:14:25.5874281Z Generating XML reports...
2023-01-11T21:14:25.5874619Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211301.xml
2023-01-11T21:14:25.5874891Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5875013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5875290Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5875423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5875606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdgcik2cl
2023-01-11T21:14:25.5875834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdgcik2cl/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5876078Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5876083Z 
2023-01-11T21:14:25.5876153Z Running tests...
2023-01-11T21:14:25.5876337Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5876591Z   test_udf_remote_message_dropped_timeout_to_self (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32301
2023-01-11T21:14:25.5876751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32302
2023-01-11T21:14:25.5876905Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32303
2023-01-11T21:14:25.5877056Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32304
2023-01-11T21:14:25.5877355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5877480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5877760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5877881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5878065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34fjeoch
2023-01-11T21:14:25.5878260Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34fjeoch/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5878532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5878652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5878927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5879063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5879245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqgo_dzmn
2023-01-11T21:14:25.5879438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqgo_dzmn/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5879695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5879821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5880098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5880231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5880413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjw51xukw
2023-01-11T21:14:25.5880611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjw51xukw/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5880879Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5881001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5881267Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5881398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5881580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpymo5_i82
2023-01-11T21:14:25.5881770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpymo5_i82/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5881932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5882094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5882295Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5882454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5882519Z ok (8.335s)
2023-01-11T21:14:25.5882524Z 
2023-01-11T21:14:25.5882710Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5882784Z Ran 1 test in 8.336s
2023-01-11T21:14:25.5882789Z 
2023-01-11T21:14:25.5882847Z OK
2023-01-11T21:14:25.5882852Z 
2023-01-11T21:14:25.5882934Z Generating XML reports...
2023-01-11T21:14:25.5883289Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211312.xml
2023-01-11T21:14:25.5883558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5883709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5883989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5884110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5884293Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpisbe8vuk
2023-01-11T21:14:25.5884485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpisbe8vuk/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5884726Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5884731Z 
2023-01-11T21:14:25.5884803Z Running tests...
2023-01-11T21:14:25.5884996Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5885229Z   test_verify_backend_options (__main__.FaultyFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32476
2023-01-11T21:14:25.5885389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32477
2023-01-11T21:14:25.5885541Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32478
2023-01-11T21:14:25.5885681Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32479
2023-01-11T21:14:25.5885952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5886076Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5886354Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5886487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5886670Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6g8gnek6
2023-01-11T21:14:25.5886866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6g8gnek6/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5887136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5887245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5887520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5887654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5887836Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpow_0rovo
2023-01-11T21:14:25.5888031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpow_0rovo/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5888305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5888427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5888741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5888873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5889042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0bh1nu1i
2023-01-11T21:14:25.5889238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0bh1nu1i/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5889508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5889631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5889909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5890043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5890260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3di1xx6
2023-01-11T21:14:25.5890457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3di1xx6/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5890621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5890768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5891094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5891257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5891325Z ok (8.341s)
2023-01-11T21:14:25.5891330Z 
2023-01-11T21:14:25.5891536Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5891613Z Ran 1 test in 8.341s
2023-01-11T21:14:25.5891618Z 
2023-01-11T21:14:25.5891681Z OK
2023-01-11T21:14:25.5891685Z 
2023-01-11T21:14:25.5891776Z Generating XML reports...
2023-01-11T21:14:25.5892113Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211322.xml
2023-01-11T21:14:25.5892386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5892511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5892793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5892931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5893116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbh6cxhgu
2023-01-11T21:14:25.5893314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbh6cxhgu/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5893560Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5893567Z 
2023-01-11T21:14:25.5893640Z Running tests...
2023-01-11T21:14:25.5893823Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5894071Z   test_remote_timeout_to_here_in_jit (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32651
2023-01-11T21:14:25.5894227Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32652
2023-01-11T21:14:25.5894380Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 32653
2023-01-11T21:14:25.5894534Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 32654
2023-01-11T21:14:25.5894806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5894929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5895211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5895394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5895578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptobqltv3
2023-01-11T21:14:25.5895774Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptobqltv3/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5896044Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5896165Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5896446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5896581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5896802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfesjc41s
2023-01-11T21:14:25.5897001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfesjc41s/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5897261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5897387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5897666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5897799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5897982Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptec9n4bp
2023-01-11T21:14:25.5898178Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptec9n4bp/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5898447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5898572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5898835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5898968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5899150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7kacoeua
2023-01-11T21:14:25.5899341Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7kacoeua/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5899506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5899670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5899826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5899988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5900058Z ok (8.341s)
2023-01-11T21:14:25.5900063Z 
2023-01-11T21:14:25.5900314Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5900391Z Ran 1 test in 8.341s
2023-01-11T21:14:25.5900396Z 
2023-01-11T21:14:25.5900456Z OK
2023-01-11T21:14:25.5900461Z 
2023-01-11T21:14:25.5900547Z Generating XML reports...
2023-01-11T21:14:25.5900914Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20230111211332.xml
2023-01-11T21:14:25.5901186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5901307Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5901585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5901707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5901933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn1gv1o7r
2023-01-11T21:14:25.5902128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn1gv1o7r/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5902374Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5902380Z 
2023-01-11T21:14:25.5902452Z Running tests...
2023-01-11T21:14:25.5902648Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5902889Z   test_rref_timeout_pickle_in_jit (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 358
2023-01-11T21:14:25.5903041Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 359
2023-01-11T21:14:25.5903189Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 360
2023-01-11T21:14:25.5903356Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 361
2023-01-11T21:14:25.5903634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5903756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5904032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5904165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5904349Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyepqa2fk
2023-01-11T21:14:25.5904615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5904738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5905017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5905141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5905325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwoptfyrf
2023-01-11T21:14:25.5905519Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyepqa2fk/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5905708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwoptfyrf/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5905977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5906097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5906374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5906505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5906678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_koqnmjc
2023-01-11T21:14:25.5906870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_koqnmjc/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5907134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5907257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5907535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5907669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5907855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8ckt0ml
2023-01-11T21:14:25.5908047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8ckt0ml/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5908244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5908394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5908553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5908708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5908774Z ok (8.338s)
2023-01-11T21:14:25.5908779Z 
2023-01-11T21:14:25.5908976Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5909049Z Ran 1 test in 8.339s
2023-01-11T21:14:25.5909054Z 
2023-01-11T21:14:25.5909113Z OK
2023-01-11T21:14:25.5909118Z 
2023-01-11T21:14:25.5909200Z Generating XML reports...
2023-01-11T21:14:25.5909551Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20230111211342.xml
2023-01-11T21:14:25.5909853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5909978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5910258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5910390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5910573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2lgc13tp
2023-01-11T21:14:25.5910767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2lgc13tp/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5911008Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5911013Z 
2023-01-11T21:14:25.5911085Z Running tests...
2023-01-11T21:14:25.5911266Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5911517Z   test_rref_timeout_pickle_script_func (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 533
2023-01-11T21:14:25.5911670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 534
2023-01-11T21:14:25.5911818Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 535
2023-01-11T21:14:25.5911967Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 536
2023-01-11T21:14:25.5912234Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5912355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5912631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5912752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5912943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp7h8oh7n
2023-01-11T21:14:25.5913136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp7h8oh7n/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5913406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5913527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5913803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5913935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5914119Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzg0iq57v
2023-01-11T21:14:25.5914310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzg0iq57v/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5914569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5914722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5914999Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5915130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5915312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn85cjodz
2023-01-11T21:14:25.5915506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn85cjodz/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5915775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5915896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5916188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5916324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5916504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3_cz8vyt
2023-01-11T21:14:25.5916697Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3_cz8vyt/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5916860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5917018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5917177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5917335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5917403Z ok (8.339s)
2023-01-11T21:14:25.5917408Z 
2023-01-11T21:14:25.5917592Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5917670Z Ran 1 test in 8.340s
2023-01-11T21:14:25.5917675Z 
2023-01-11T21:14:25.5917737Z OK
2023-01-11T21:14:25.5917742Z 
2023-01-11T21:14:25.5917824Z Generating XML reports...
2023-01-11T21:14:25.5918186Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20230111211352.xml
2023-01-11T21:14:25.5918456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5918576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5918852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5918973Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5919155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbapyptaq
2023-01-11T21:14:25.5919352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbapyptaq/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5919598Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5919602Z 
2023-01-11T21:14:25.5919674Z Running tests...
2023-01-11T21:14:25.5919872Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5920113Z   test_rref_to_here_timeout_in_jit (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 710
2023-01-11T21:14:25.5920265Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 711
2023-01-11T21:14:25.5920414Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 712
2023-01-11T21:14:25.5920551Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 713
2023-01-11T21:14:25.5920824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5920978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5921255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5921389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5921573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptrzmk5u9
2023-01-11T21:14:25.5921768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptrzmk5u9/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5922038Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5922159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5922420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5922584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5922770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5aqxixt
2023-01-11T21:14:25.5922963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5aqxixt/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5923231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5923352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5923627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5923759Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5923930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn8shogm1
2023-01-11T21:14:25.5924125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn8shogm1/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5924395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5924514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5924791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5924923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5925105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3sb2d_n
2023-01-11T21:14:25.5925297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3sb2d_n/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5925459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5925608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5925773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5925930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5925996Z ok (4.425s)
2023-01-11T21:14:25.5926001Z 
2023-01-11T21:14:25.5926195Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5926269Z Ran 1 test in 4.425s
2023-01-11T21:14:25.5926274Z 
2023-01-11T21:14:25.5926334Z OK
2023-01-11T21:14:25.5926338Z 
2023-01-11T21:14:25.5926421Z Generating XML reports...
2023-01-11T21:14:25.5926768Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20230111211402.xml
2023-01-11T21:14:25.5927037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5927158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5927482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5927614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5927796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcjddbfl4
2023-01-11T21:14:25.5927989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcjddbfl4/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5928234Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5928239Z 
2023-01-11T21:14:25.5928311Z Running tests...
2023-01-11T21:14:25.5928492Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5928725Z   test_timeout_in_python (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 885
2023-01-11T21:14:25.5928904Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 886
2023-01-11T21:14:25.5929056Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 887
2023-01-11T21:14:25.5929206Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 888
2023-01-11T21:14:25.5929479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5929602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5929882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5930004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5930191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjit_t18v
2023-01-11T21:14:25.5930386Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjit_t18v/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5930659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5930782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5931258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5931393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5931577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr1_b44__
2023-01-11T21:14:25.5931769Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr1_b44__/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5932027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5932149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5932430Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5932571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5932753Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qy9jjg4
2023-01-11T21:14:25.5932946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qy9jjg4/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5933214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5933336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5933614Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5933736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5933920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprktdpct6
2023-01-11T21:14:25.5934173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprktdpct6/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5934339Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5934498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5934656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5934817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5934887Z ok (7.030s)
2023-01-11T21:14:25.5934892Z 
2023-01-11T21:14:25.5935076Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5935152Z Ran 1 test in 7.031s
2023-01-11T21:14:25.5935158Z 
2023-01-11T21:14:25.5935219Z OK
2023-01-11T21:14:25.5935224Z 
2023-01-11T21:14:25.5935308Z Generating XML reports...
2023-01-11T21:14:25.5935705Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20230111211409.xml
2023-01-11T21:14:25.5935983Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5936104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5936383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5936516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5936687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5gb4n8pe
2023-01-11T21:14:25.5936882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5gb4n8pe/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5937120Z Test results will be stored in test-reports/python-unittest/distributed.rpc.test_faulty_agent
2023-01-11T21:14:25.5937124Z 
2023-01-11T21:14:25.5937200Z Running tests...
2023-01-11T21:14:25.5937396Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5937641Z   test_timeout_in_torchscript_function (__main__.FaultyJitFaultyAgentRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1060
2023-01-11T21:14:25.5937794Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1061
2023-01-11T21:14:25.5937944Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1062
2023-01-11T21:14:25.5938082Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1063
2023-01-11T21:14:25.5938351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5938473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5938754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5938892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5939077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeg2ewn9w
2023-01-11T21:14:25.5939273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeg2ewn9w/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5939544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5939663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5939926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5940058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5940305Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd__599vs
2023-01-11T21:14:25.5940500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd__599vs/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5940806Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5940929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5941208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5941340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5941510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxcb43tdn
2023-01-11T21:14:25.5941704Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxcb43tdn/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5941974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:14:25.5942136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:14:25.5942419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:14:25.5942550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:14:25.5942733Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpigjiye0_
2023-01-11T21:14:25.5942926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpigjiye0_/_remote_module_non_scriptable.py
2023-01-11T21:14:25.5943090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:14:25.5943239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:14:25.5943402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:14:25.5943564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:14:25.5943637Z ok (6.936s)
2023-01-11T21:14:25.5943642Z 
2023-01-11T21:14:25.5943836Z ----------------------------------------------------------------------
2023-01-11T21:14:25.5943913Z Ran 1 test in 6.936s
2023-01-11T21:14:25.5943918Z 
2023-01-11T21:14:25.5943978Z OK
2023-01-11T21:14:25.5943983Z 
2023-01-11T21:14:25.5944068Z Generating XML reports...
2023-01-11T21:14:25.5944416Z Generated XML report: test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20230111211418.xml
2023-01-11T21:14:25.5944436Z 
2023-01-11T21:14:25.5944772Z ##[endgroup]
2023-01-11T21:14:25.5945120Z FINISHED PRINTING LOG FILE of distributed/rpc/test_faulty_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_faulty_agent_32hjgg0g)
2023-01-11T21:14:25.5945126Z 
2023-01-11T21:14:25.5945337Z Running distributed/rpc/cuda/test_tensorpipe_agent ... [2023-01-11 21:14:25.542510]
2023-01-11T21:14:25.5945741Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:14:25.543033]
2023-01-11T21:21:28.0133657Z 
2023-01-11T21:21:28.0134083Z Expand the folded group to see the log file of distributed/rpc/cuda/test_tensorpipe_agent
2023-01-11T21:21:28.0134909Z ##[group]PRINTING LOG FILE of distributed/rpc/cuda/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-cuda-test_tensorpipe_agent_ezxa9svg)
2023-01-11T21:21:28.0139144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps_h5n24w
2023-01-11T21:21:28.0139831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps_h5n24w/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0141079Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaDdpComparisonTest testMethod=test_ddp_dist_autograd_local_vs_remote_gpu>]>
2023-01-11T21:21:28.0141819Z test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest)
2023-01-11T21:21:28.0143053Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_simple>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation_gpu_root>]>
2023-01-11T21:21:28.0143924Z test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest)
2023-01-11T21:21:28.0144393Z test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest)
2023-01-11T21:21:28.0212058Z test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest)
2023-01-11T21:21:28.0213325Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device_script>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_invalid_devices>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_valid_device>]>
2023-01-11T21:21:28.0214881Z test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest)
2023-01-11T21:21:28.0215478Z test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest)
2023-01-11T21:21:28.0216004Z test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest)
2023-01-11T21:21:28.0216513Z test_valid_device (__main__.TensorPipeCudaRemoteModuleTest)
2023-01-11T21:21:28.0217093Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaRpcTest testMethod=test_profiler_remote_cuda>]>
2023-01-11T21:21:28.0217646Z test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest)
2023-01-11T21:21:28.0219222Z <unittest.suite.TestSuite tests=[<__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never_find_unused>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never_find_unused>]>
2023-01-11T21:21:28.0220796Z test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest)
2023-01-11T21:21:28.0221284Z test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest)
2023-01-11T21:21:28.0221870Z test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest)
2023-01-11T21:21:28.0222397Z test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest)
2023-01-11T21:21:28.0222917Z test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest)
2023-01-11T21:21:28.0223403Z test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest)
2023-01-11T21:21:28.0223900Z test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest)
2023-01-11T21:21:28.0224410Z test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest)
2023-01-11T21:21:28.0255660Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_nested_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_callback_changes_devices>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_int>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_str>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_not_cuda>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_modify_tensor_inplace>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_replace_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_value_on_bad_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default_to_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default_to_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_in_options>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_local_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_remote_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_min_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_many_to_one>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_not_timeout>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_one_to_many>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_wrong_worker_name>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch_reverse>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_with_unpickleable_attributes>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_tensor_view_as_return_value>]>
2023-01-11T21:21:28.0270629Z test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0271030Z test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0271419Z test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0271820Z test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0272197Z test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0272607Z test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0273038Z test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0273500Z test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0273893Z test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0274278Z test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0274641Z test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0274992Z test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0275355Z test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0275728Z test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0276100Z test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0276487Z test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0276843Z test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0277184Z test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0277518Z test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0277867Z test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0278218Z test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0278578Z test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0278938Z test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0279304Z test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0279677Z test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0280033Z test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0280388Z test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0280735Z test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0281080Z test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0281411Z test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0281760Z test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0282104Z test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0282436Z test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0282795Z test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0283157Z test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0283507Z test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0283870Z test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0284227Z test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0284584Z test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0284926Z test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0285279Z test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0285637Z test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0285997Z test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0286375Z test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0286798Z test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0287159Z test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0287494Z test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0287863Z test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0288246Z test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0288617Z test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0288979Z test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0289322Z test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0289689Z test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0290094Z test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0290473Z test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0290964Z test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0291355Z test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0291743Z test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0292114Z test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0292457Z test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0292814Z test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0293166Z test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0293508Z test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0293873Z test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0294244Z test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0294583Z test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0294931Z test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0295293Z test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0295671Z test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0296046Z test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0296426Z test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0296805Z test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0297169Z test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0297535Z test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0297901Z test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0298264Z test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0298618Z test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0298986Z test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0299353Z test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0299718Z test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0300069Z test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0300497Z test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0300864Z test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0301218Z test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0301643Z test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0302018Z test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0302389Z test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T21:21:28.0303112Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_device_maps_backward_pass>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_dist_autograd_sync_streams>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_gradients_synchronizations>]>
2023-01-11T21:21:28.0303802Z test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2023-01-11T21:21:28.0304184Z test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2023-01-11T21:21:28.0304572Z test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2023-01-11T21:21:28.0305144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0305478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0305910Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0306242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0306691Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6s3d2wva
2023-01-11T21:21:28.0307091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6s3d2wva/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0307575Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0307791Z 
2023-01-11T21:21:28.0307850Z Running tests...
2023-01-11T21:21:28.0308150Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0308551Z   test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest) ... skip: c10d was not compiled with the NCCL backend (0.002s)
2023-01-11T21:21:28.0308804Z 
2023-01-11T21:21:28.0308996Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0309218Z Ran 1 test in 0.002s
2023-01-11T21:21:28.0309328Z 
2023-01-11T21:21:28.0309399Z OK (skipped=1)
2023-01-11T21:21:28.0309505Z 
2023-01-11T21:21:28.0309588Z Generating XML reports...
2023-01-11T21:21:28.0310096Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20230111211428.xml
2023-01-11T21:21:28.0310675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0310999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0311423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0311751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0312089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz_yhu9mf
2023-01-11T21:21:28.0312487Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz_yhu9mf/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0312953Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0313168Z 
2023-01-11T21:21:28.0313238Z Running tests...
2023-01-11T21:21:28.0313583Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0313983Z   test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1259
2023-01-11T21:21:28.0314366Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1260
2023-01-11T21:21:28.0314690Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1261
2023-01-11T21:21:28.0315011Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1262
2023-01-11T21:21:28.0315463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0315778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0316207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0316584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0316919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvu8161s0
2023-01-11T21:21:28.0317319Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvu8161s0/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0317807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0318137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0318551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0318893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0319236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytay5qqg
2023-01-11T21:21:28.0319622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytay5qqg/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0320115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0320445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0320869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0321199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0321540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8173x1qu
2023-01-11T21:21:28.0321934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8173x1qu/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0322417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0322728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0323155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0323501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0323829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk4seflu8
2023-01-11T21:21:28.0324229Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk4seflu8/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0324603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0324946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0325278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0325614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0325896Z skip: Need at least 1 CUDA device (2.233s)
2023-01-11T21:21:28.0326071Z 
2023-01-11T21:21:28.0326257Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0326494Z Ran 1 test in 2.233s
2023-01-11T21:21:28.0326608Z 
2023-01-11T21:21:28.0328875Z OK (skipped=1)
2023-01-11T21:21:28.0328988Z 
2023-01-11T21:21:28.0329073Z Generating XML reports...
2023-01-11T21:21:28.0329593Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20230111211430.xml
2023-01-11T21:21:28.0330168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0330495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0331065Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0331397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0331821Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvmsn4ck_
2023-01-11T21:21:28.0332222Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvmsn4ck_/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0332690Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0332902Z 
2023-01-11T21:21:28.0332975Z Running tests...
2023-01-11T21:21:28.0333275Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0333682Z   test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1314
2023-01-11T21:21:28.0334084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1315
2023-01-11T21:21:28.0334408Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1316
2023-01-11T21:21:28.0334739Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1317
2023-01-11T21:21:28.0335172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0335497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0335916Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0336255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0336582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsrxodhte
2023-01-11T21:21:28.0336983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsrxodhte/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0337463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0337786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0338201Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0338539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0338883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdnset22t
2023-01-11T21:21:28.0339263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdnset22t/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0339740Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0340063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0340485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0340810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0341148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1u1ukq4f
2023-01-11T21:21:28.0341646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1u1ukq4f/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0342137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0342446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0342867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0343211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0343538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi0wwwmxr
2023-01-11T21:21:28.0343933Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi0wwwmxr/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0344341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0344689Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0345019Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0345354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0345630Z skip: Need at least 1 CUDA device (2.330s)
2023-01-11T21:21:28.0345765Z 
2023-01-11T21:21:28.0345954Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0346189Z Ran 1 test in 2.330s
2023-01-11T21:21:28.0346299Z 
2023-01-11T21:21:28.0346371Z OK (skipped=1)
2023-01-11T21:21:28.0346478Z 
2023-01-11T21:21:28.0346560Z Generating XML reports...
2023-01-11T21:21:28.0347058Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20230111211434.xml
2023-01-11T21:21:28.0347634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0347960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0348372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0348713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0349051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgx8tbxe3
2023-01-11T21:21:28.0349444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgx8tbxe3/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0349906Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0350122Z 
2023-01-11T21:21:28.0350193Z Running tests...
2023-01-11T21:21:28.0350490Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0350912Z   test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1369
2023-01-11T21:21:28.0351319Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1370
2023-01-11T21:21:28.0351642Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1371
2023-01-11T21:21:28.0351962Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1372
2023-01-11T21:21:28.0352394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0352719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0353140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0353480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0353809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyv_te4v9
2023-01-11T21:21:28.0354241Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyv_te4v9/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0354721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0355031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0355453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0355787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0356123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkyn8xjjx
2023-01-11T21:21:28.0356505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkyn8xjjx/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0357015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0357341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0357762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0358086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0358425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqordqqg8
2023-01-11T21:21:28.0358821Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqordqqg8/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0359287Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0359608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0360031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0360375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0360701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyfe5y8mz
2023-01-11T21:21:28.0361094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyfe5y8mz/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0361465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0361797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0362138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0362478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0362756Z skip: Need at least 1 CUDA device (2.235s)
2023-01-11T21:21:28.0362888Z 
2023-01-11T21:21:28.0363075Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0363312Z Ran 1 test in 2.235s
2023-01-11T21:21:28.0363422Z 
2023-01-11T21:21:28.0363493Z OK (skipped=1)
2023-01-11T21:21:28.0363602Z 
2023-01-11T21:21:28.0363673Z Generating XML reports...
2023-01-11T21:21:28.0364180Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20230111211438.xml
2023-01-11T21:21:28.0364754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0365078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0365488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0365825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0366160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbnkoam7f
2023-01-11T21:21:28.0366590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbnkoam7f/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0367050Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0367264Z 
2023-01-11T21:21:28.0367335Z Running tests...
2023-01-11T21:21:28.0367633Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0368033Z   test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1424
2023-01-11T21:21:28.0368451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1425
2023-01-11T21:21:28.0368894Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0369223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0369664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0370013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0370358Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx1m5k8eg
2023-01-11T21:21:28.0370759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx1m5k8eg/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0371346Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0371677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0372105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0372437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0372780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_ns3npo
2023-01-11T21:21:28.0373182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_ns3npo/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0373555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0373889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0374170Z skip: Need at least 1 CUDA device (2.126s)
2023-01-11T21:21:28.0374304Z 
2023-01-11T21:21:28.0374503Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0374731Z Ran 1 test in 2.126s
2023-01-11T21:21:28.0374844Z 
2023-01-11T21:21:28.0374917Z OK (skipped=1)
2023-01-11T21:21:28.0375024Z 
2023-01-11T21:21:28.0375110Z Generating XML reports...
2023-01-11T21:21:28.0375620Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111211442.xml
2023-01-11T21:21:28.0376193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0376522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0376942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0377287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0377615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3wt3pb2
2023-01-11T21:21:28.0378015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3wt3pb2/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0378491Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0378704Z 
2023-01-11T21:21:28.0378762Z Running tests...
2023-01-11T21:21:28.0379065Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0379562Z   test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1457
2023-01-11T21:21:28.0379986Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1458
2023-01-11T21:21:28.0380421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0380750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0381178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0381506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0381915Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48i2mle_
2023-01-11T21:21:28.0382313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48i2mle_/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0382843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0383155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0383579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0383918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0384256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7mvutrjk
2023-01-11T21:21:28.0384637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7mvutrjk/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0385008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0385348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0385619Z skip: Need at least 1 CUDA device (2.118s)
2023-01-11T21:21:28.0385754Z 
2023-01-11T21:21:28.0385951Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0386188Z Ran 1 test in 2.119s
2023-01-11T21:21:28.0386298Z 
2023-01-11T21:21:28.0386368Z OK (skipped=1)
2023-01-11T21:21:28.0386462Z 
2023-01-11T21:21:28.0386544Z Generating XML reports...
2023-01-11T21:21:28.0387051Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111211446.xml
2023-01-11T21:21:28.0387623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0387936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0388355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0388696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0389037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3vh8yzsq
2023-01-11T21:21:28.0389420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3vh8yzsq/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0389899Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0390111Z 
2023-01-11T21:21:28.0390181Z Running tests...
2023-01-11T21:21:28.0390468Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0390873Z   test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1490
2023-01-11T21:21:28.0391274Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1491
2023-01-11T21:21:28.0391725Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0392072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0392493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0392833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0393173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0u7w429a
2023-01-11T21:21:28.0393558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0u7w429a/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0394034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0394356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0394766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0395139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0395483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzkuiihe
2023-01-11T21:21:28.0395880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzkuiihe/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0396239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0396581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0396863Z skip: Need at least 1 CUDA device (2.119s)
2023-01-11T21:21:28.0396995Z 
2023-01-11T21:21:28.0397195Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0397422Z Ran 1 test in 2.119s
2023-01-11T21:21:28.0397533Z 
2023-01-11T21:21:28.0397604Z OK (skipped=1)
2023-01-11T21:21:28.0397709Z 
2023-01-11T21:21:28.0397793Z Generating XML reports...
2023-01-11T21:21:28.0398290Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111211450.xml
2023-01-11T21:21:28.0398867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0399188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0399608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0399934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0400269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1alezwam
2023-01-11T21:21:28.0400665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1alezwam/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0401128Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0401340Z 
2023-01-11T21:21:28.0401416Z Running tests...
2023-01-11T21:21:28.0401712Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0402115Z   test_valid_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1523
2023-01-11T21:21:28.0402505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1524
2023-01-11T21:21:28.0402948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0403274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0403685Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0404025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0404365Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0xdn0an
2023-01-11T21:21:28.0404800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0xdn0an/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0405270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0405591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0406013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0406353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0406679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp857xkad3
2023-01-11T21:21:28.0407070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp857xkad3/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0407442Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0407802Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0408085Z skip: Need at least 1 CUDA device (2.121s)
2023-01-11T21:21:28.0408218Z 
2023-01-11T21:21:28.0408417Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0408652Z Ran 1 test in 2.121s
2023-01-11T21:21:28.0408749Z 
2023-01-11T21:21:28.0408821Z OK (skipped=1)
2023-01-11T21:21:28.0408926Z 
2023-01-11T21:21:28.0409009Z Generating XML reports...
2023-01-11T21:21:28.0409513Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111211454.xml
2023-01-11T21:21:28.0410068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0410392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0410815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0411281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0411610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3wdi05g5
2023-01-11T21:21:28.0412005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3wdi05g5/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0412480Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0412695Z 
2023-01-11T21:21:28.0412767Z Running tests...
2023-01-11T21:21:28.0413052Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0413451Z   test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1556
2023-01-11T21:21:28.0413845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1557
2023-01-11T21:21:28.0414160Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1558
2023-01-11T21:21:28.0414489Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1559
2023-01-11T21:21:28.0414935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0415260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0415665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0416006Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0416345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpajr8j6a3
2023-01-11T21:21:28.0416730Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpajr8j6a3/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0417213Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0417595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0418022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0418348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0418689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw66y_l4n
2023-01-11T21:21:28.0419088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw66y_l4n/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0419571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0419882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0420302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0420683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0421010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmul9rwvv
2023-01-11T21:21:28.0421404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmul9rwvv/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0421965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0422291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0422701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0423040Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0423380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7mtumsck
2023-01-11T21:21:28.0423767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7mtumsck/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0424145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0424490Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0424837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0425169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0425453Z skip: Need at least 2 CUDA devices (2.331s)
2023-01-11T21:21:28.0425591Z 
2023-01-11T21:21:28.0425790Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0426032Z Ran 1 test in 2.331s
2023-01-11T21:21:28.0426130Z 
2023-01-11T21:21:28.0426204Z OK (skipped=1)
2023-01-11T21:21:28.0426313Z 
2023-01-11T21:21:28.0426400Z Generating XML reports...
2023-01-11T21:21:28.0426887Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20230111211458.xml
2023-01-11T21:21:28.0427429Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0427758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0428180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0428524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0428851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7mvbk2j
2023-01-11T21:21:28.0429248Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7mvbk2j/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0429725Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0429939Z 
2023-01-11T21:21:28.0429998Z Running tests...
2023-01-11T21:21:28.0430347Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0430748Z   test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1611
2023-01-11T21:21:28.0431151Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1612
2023-01-11T21:21:28.0431584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0431915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0432345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0432691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0433021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_icb8zob
2023-01-11T21:21:28.0433447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_icb8zob/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0433933Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0434244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0434666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0435004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0435345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj3g42eel
2023-01-11T21:21:28.0435731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj3g42eel/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0436102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0436446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0436722Z skip: Need at least 4 CUDA devices (2.125s)
2023-01-11T21:21:28.0436856Z 
2023-01-11T21:21:28.0437052Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0437287Z Ran 1 test in 2.125s
2023-01-11T21:21:28.0437398Z 
2023-01-11T21:21:28.0437471Z OK (skipped=1)
2023-01-11T21:21:28.0437565Z 
2023-01-11T21:21:28.0437646Z Generating XML reports...
2023-01-11T21:21:28.0438138Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111211502.xml
2023-01-11T21:21:28.0438702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0439014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0439440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0439789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0440127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_bx97v0e
2023-01-11T21:21:28.0440507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_bx97v0e/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0440982Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0441195Z 
2023-01-11T21:21:28.0441266Z Running tests...
2023-01-11T21:21:28.0441561Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0441958Z   test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1644
2023-01-11T21:21:28.0442367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1645
2023-01-11T21:21:28.0442819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0443177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0443601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0443939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0444278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppa8vtlaz
2023-01-11T21:21:28.0444666Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppa8vtlaz/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0445147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0445468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0445907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0446247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0446581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpya_bior_
2023-01-11T21:21:28.0446971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpya_bior_/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0447331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0447676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0447957Z skip: Need at least 4 CUDA devices (2.125s)
2023-01-11T21:21:28.0448093Z 
2023-01-11T21:21:28.0448291Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0448515Z Ran 1 test in 2.125s
2023-01-11T21:21:28.0448624Z 
2023-01-11T21:21:28.0448694Z OK (skipped=1)
2023-01-11T21:21:28.0448797Z 
2023-01-11T21:21:28.0448880Z Generating XML reports...
2023-01-11T21:21:28.0449366Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111211506.xml
2023-01-11T21:21:28.0449930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0450257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0450678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0451129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0451472Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfgz9smb5
2023-01-11T21:21:28.0451868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfgz9smb5/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0452335Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0452553Z 
2023-01-11T21:21:28.0452624Z Running tests...
2023-01-11T21:21:28.0452925Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0453325Z   test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1677
2023-01-11T21:21:28.0453715Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1678
2023-01-11T21:21:28.0454161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0454488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0454908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0455235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0455578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpag6x2qrx
2023-01-11T21:21:28.0456038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpag6x2qrx/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0456511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0456835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0457265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0457607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0457933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwmomiskh
2023-01-11T21:21:28.0458331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwmomiskh/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0458753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0459087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0459368Z skip: Need at least 4 CUDA devices (2.119s)
2023-01-11T21:21:28.0459499Z 
2023-01-11T21:21:28.0459695Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0459934Z Ran 1 test in 2.119s
2023-01-11T21:21:28.0460031Z 
2023-01-11T21:21:28.0460102Z OK (skipped=1)
2023-01-11T21:21:28.0460207Z 
2023-01-11T21:21:28.0460290Z Generating XML reports...
2023-01-11T21:21:28.0460785Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111211510.xml
2023-01-11T21:21:28.0461338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0461739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0462170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0462516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0462845Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptorcgkws
2023-01-11T21:21:28.0463243Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptorcgkws/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0463722Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0463937Z 
2023-01-11T21:21:28.0464008Z Running tests...
2023-01-11T21:21:28.0464293Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0464699Z   test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1710
2023-01-11T21:21:28.0465111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1711
2023-01-11T21:21:28.0465551Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0465877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0466304Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0466646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0466971Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd_h_fb1e
2023-01-11T21:21:28.0467368Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd_h_fb1e/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0467851Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0468182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0468657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0468998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0469338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpycvj6ac7
2023-01-11T21:21:28.0469727Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpycvj6ac7/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0470105Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0470451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0470735Z skip: Need at least 4 CUDA devices (2.119s)
2023-01-11T21:21:28.0470856Z 
2023-01-11T21:21:28.0471054Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0471292Z Ran 1 test in 2.119s
2023-01-11T21:21:28.0471404Z 
2023-01-11T21:21:28.0471510Z OK (skipped=1)
2023-01-11T21:21:28.0471617Z 
2023-01-11T21:21:28.0471687Z Generating XML reports...
2023-01-11T21:21:28.0472177Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111211514.xml
2023-01-11T21:21:28.0472742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0473064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0473473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0473815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0474152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5kwca69m
2023-01-11T21:21:28.0474541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5kwca69m/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0475023Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0475237Z 
2023-01-11T21:21:28.0475308Z Running tests...
2023-01-11T21:21:28.0475603Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0475963Z   test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:21:28.0476188Z 
2023-01-11T21:21:28.0476381Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0476623Z Ran 1 test in 0.001s
2023-01-11T21:21:28.0476732Z 
2023-01-11T21:21:28.0476803Z OK (skipped=1)
2023-01-11T21:21:28.0476899Z 
2023-01-11T21:21:28.0476983Z Generating XML reports...
2023-01-11T21:21:28.0477470Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111211518.xml
2023-01-11T21:21:28.0478048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0478370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0478796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0479138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0479481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnapmkxa5
2023-01-11T21:21:28.0479872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnapmkxa5/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0480352Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0480568Z 
2023-01-11T21:21:28.0480645Z Running tests...
2023-01-11T21:21:28.0480935Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0481356Z   test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:21:28.0481590Z 
2023-01-11T21:21:28.0481786Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0482025Z Ran 1 test in 0.000s
2023-01-11T21:21:28.0482137Z 
2023-01-11T21:21:28.0482196Z OK (skipped=1)
2023-01-11T21:21:28.0482302Z 
2023-01-11T21:21:28.0482392Z Generating XML reports...
2023-01-11T21:21:28.0482883Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111211520.xml
2023-01-11T21:21:28.0483446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0483761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0484222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0484568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0484898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9g5n3t1t
2023-01-11T21:21:28.0485296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9g5n3t1t/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0485775Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0485991Z 
2023-01-11T21:21:28.0486066Z Running tests...
2023-01-11T21:21:28.0486354Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0486722Z   test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:21:28.0486946Z 
2023-01-11T21:21:28.0487140Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0487377Z Ran 1 test in 0.000s
2023-01-11T21:21:28.0487478Z 
2023-01-11T21:21:28.0487549Z OK (skipped=1)
2023-01-11T21:21:28.0487656Z 
2023-01-11T21:21:28.0487742Z Generating XML reports...
2023-01-11T21:21:28.0488232Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111211521.xml
2023-01-11T21:21:28.0488779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0489107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0489529Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0489873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0490205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxx4zm04n
2023-01-11T21:21:28.0490607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxx4zm04n/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0491201Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0491411Z 
2023-01-11T21:21:28.0491471Z Running tests...
2023-01-11T21:21:28.0491771Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0492154Z   test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... skip: c10d was not compiled with the NCCL backend (0.000s)
2023-01-11T21:21:28.0492389Z 
2023-01-11T21:21:28.0492583Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0492801Z Ran 1 test in 0.000s
2023-01-11T21:21:28.0492910Z 
2023-01-11T21:21:28.0492980Z OK (skipped=1)
2023-01-11T21:21:28.0493085Z 
2023-01-11T21:21:28.0493168Z Generating XML reports...
2023-01-11T21:21:28.0493648Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111211523.xml
2023-01-11T21:21:28.0494262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0494589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0495010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0495341Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0495685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0bf952ja
2023-01-11T21:21:28.0496083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0bf952ja/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0496564Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0496766Z 
2023-01-11T21:21:28.0496839Z Running tests...
2023-01-11T21:21:28.0497184Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0497631Z   test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1783
2023-01-11T21:21:28.0498056Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1784
2023-01-11T21:21:28.0498380Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1785
2023-01-11T21:21:28.0498701Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1786
2023-01-11T21:21:28.0499335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0499823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0500538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0501056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0501409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmpka27aa
2023-01-11T21:21:28.0501880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmpka27aa/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0502385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0502714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0503130Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0503476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0503818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8av2ni3
2023-01-11T21:21:28.0504213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8av2ni3/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0504687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0505008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0505436Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0505765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0506105Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjrelvlq
2023-01-11T21:21:28.0506501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjrelvlq/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0506981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0507292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0507717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0508143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0508485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsiezx6yt
2023-01-11T21:21:28.0508874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsiezx6yt/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0509247Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0509598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0509930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0510272Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0510550Z skip: Need at least 1 CUDA device (2.330s)
2023-01-11T21:21:28.0510720Z 
2023-01-11T21:21:28.0510919Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0511143Z Ran 1 test in 2.331s
2023-01-11T21:21:28.0511254Z 
2023-01-11T21:21:28.0511324Z OK (skipped=1)
2023-01-11T21:21:28.0511430Z 
2023-01-11T21:21:28.0511513Z Generating XML reports...
2023-01-11T21:21:28.0512035Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211525.xml
2023-01-11T21:21:28.0512631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0512956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0513380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0513711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0514054Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp54q7hum0
2023-01-11T21:21:28.0514446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp54q7hum0/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0514923Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0515124Z 
2023-01-11T21:21:28.0515196Z Running tests...
2023-01-11T21:21:28.0515497Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0515936Z   test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1838
2023-01-11T21:21:28.0516361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1839
2023-01-11T21:21:28.0516689Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1840
2023-01-11T21:21:28.0517012Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1841
2023-01-11T21:21:28.0517461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0517777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0518197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0518539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0518870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8epze5fc
2023-01-11T21:21:28.0519263Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8epze5fc/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0519747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0520071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0520518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0520857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0521190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmporr7giol
2023-01-11T21:21:28.0521584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmporr7giol/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0522051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0522371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0522794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0523120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0523486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmbnhr8ln
2023-01-11T21:21:28.0523885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmbnhr8ln/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0524368Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0524674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0525096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0525433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0525757Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa3_54oi3
2023-01-11T21:21:28.0526150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa3_54oi3/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0526523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0526865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0527194Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0527532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0527810Z skip: Need at least 1 CUDA device (2.229s)
2023-01-11T21:21:28.0527946Z 
2023-01-11T21:21:28.0528146Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0528369Z Ran 1 test in 2.230s
2023-01-11T21:21:28.0528478Z 
2023-01-11T21:21:28.0528550Z OK (skipped=1)
2023-01-11T21:21:28.0528656Z 
2023-01-11T21:21:28.0528742Z Generating XML reports...
2023-01-11T21:21:28.0529264Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211529.xml
2023-01-11T21:21:28.0529868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0530192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0530615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0531118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0531462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9wf_9idx
2023-01-11T21:21:28.0531855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9wf_9idx/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0532325Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0532540Z 
2023-01-11T21:21:28.0532614Z Running tests...
2023-01-11T21:21:28.0532916Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0533428Z   test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1893
2023-01-11T21:21:28.0533861Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1894
2023-01-11T21:21:28.0534191Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1895
2023-01-11T21:21:28.0534519Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1896
2023-01-11T21:21:28.0534974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0535295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0535724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0536071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0536444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__r_irsc
2023-01-11T21:21:28.0536842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__r_irsc/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0537330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0537658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0538071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0538417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0538762Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptsynxdvj
2023-01-11T21:21:28.0539152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptsynxdvj/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0539639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0539975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0540401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0540734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0541076Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__3l_22r
2023-01-11T21:21:28.0541473Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__3l_22r/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0542015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0542335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0542763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0543109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0543434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3iqbwjo9
2023-01-11T21:21:28.0543832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3iqbwjo9/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0544215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0544558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0544887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0545221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0545509Z skip: Need at least 2 CUDA devices (2.335s)
2023-01-11T21:21:28.0545643Z 
2023-01-11T21:21:28.0545830Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0546108Z Ran 1 test in 2.336s
2023-01-11T21:21:28.0546219Z 
2023-01-11T21:21:28.0546290Z OK (skipped=1)
2023-01-11T21:21:28.0546398Z 
2023-01-11T21:21:28.0546480Z Generating XML reports...
2023-01-11T21:21:28.0547004Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211533.xml
2023-01-11T21:21:28.0547601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0547926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0548349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0548676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0549046Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph82cm09w
2023-01-11T21:21:28.0549446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph82cm09w/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0549909Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0550124Z 
2023-01-11T21:21:28.0550198Z Running tests...
2023-01-11T21:21:28.0550496Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0550943Z   test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1948
2023-01-11T21:21:28.0551371Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1949
2023-01-11T21:21:28.0551697Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 1950
2023-01-11T21:21:28.0552018Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 1951
2023-01-11T21:21:28.0552452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0552784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0553206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0553551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0553875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0x7vkyno
2023-01-11T21:21:28.0554269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0x7vkyno/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0554750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0555073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0555487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0555828Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0556166Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb74yyrlw
2023-01-11T21:21:28.0556550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb74yyrlw/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0557036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0557358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0557780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0558105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0558449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp26ulghc1
2023-01-11T21:21:28.0558897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp26ulghc1/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0559364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0559690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0560111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0560447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0560774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpos9dkp6y
2023-01-11T21:21:28.0561174Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpos9dkp6y/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0561551Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0561928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0562260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0562592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0562873Z skip: Need at least 1 CUDA device (2.231s)
2023-01-11T21:21:28.0563003Z 
2023-01-11T21:21:28.0563191Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0563431Z Ran 1 test in 2.231s
2023-01-11T21:21:28.0563543Z 
2023-01-11T21:21:28.0563617Z OK (skipped=1)
2023-01-11T21:21:28.0563721Z 
2023-01-11T21:21:28.0563804Z Generating XML reports...
2023-01-11T21:21:28.0564323Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211537.xml
2023-01-11T21:21:28.0564919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0565244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0565655Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0565994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0566334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy98jyrmc
2023-01-11T21:21:28.0566734Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy98jyrmc/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0567201Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0567413Z 
2023-01-11T21:21:28.0567485Z Running tests...
2023-01-11T21:21:28.0567783Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0568225Z   test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2003
2023-01-11T21:21:28.0568652Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2004
2023-01-11T21:21:28.0568975Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2005
2023-01-11T21:21:28.0569294Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2006
2023-01-11T21:21:28.0569724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0570057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0570480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0570948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0571301Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx802ubvt
2023-01-11T21:21:28.0571760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx802ubvt/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0572248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0572562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0572989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0573331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0573673Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplfzw_1l8
2023-01-11T21:21:28.0574056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplfzw_1l8/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0574573Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0574899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0575319Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0575645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0575981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbnky5v_9
2023-01-11T21:21:28.0576372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbnky5v_9/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0576840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0577164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0577585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0577932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0578258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmoqux8zc
2023-01-11T21:21:28.0578654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmoqux8zc/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0579024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0579353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0579692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0580030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0580315Z skip: Need at least 1 CUDA device (2.227s)
2023-01-11T21:21:28.0580446Z 
2023-01-11T21:21:28.0580632Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0580881Z Ran 1 test in 2.227s
2023-01-11T21:21:28.0580991Z 
2023-01-11T21:21:28.0581062Z OK (skipped=1)
2023-01-11T21:21:28.0581166Z 
2023-01-11T21:21:28.0581238Z Generating XML reports...
2023-01-11T21:21:28.0581845Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211541.xml
2023-01-11T21:21:28.0582450Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0582778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0583188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0583531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0583872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxyxdual6
2023-01-11T21:21:28.0584312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxyxdual6/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0584779Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0584993Z 
2023-01-11T21:21:28.0585066Z Running tests...
2023-01-11T21:21:28.0585364Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0585813Z   test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2058
2023-01-11T21:21:28.0586275Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2059
2023-01-11T21:21:28.0586601Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2060
2023-01-11T21:21:28.0586923Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2061
2023-01-11T21:21:28.0587383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0587710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0588135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0588479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0588809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpav_i3jl5
2023-01-11T21:21:28.0589208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpav_i3jl5/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0589694Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0590009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0590440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0590789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0591132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq6z3au3d
2023-01-11T21:21:28.0591522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq6z3au3d/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0592007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0592344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0592771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0593097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0593437Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgbwtpi5r
2023-01-11T21:21:28.0593842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgbwtpi5r/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0594314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0594644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0595071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0595415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0595741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxbf7sokx
2023-01-11T21:21:28.0596143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxbf7sokx/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0596518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0596892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0597243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0597583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0597862Z skip: Need at least 1 CUDA device (2.329s)
2023-01-11T21:21:28.0597981Z 
2023-01-11T21:21:28.0598183Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0598420Z Ran 1 test in 2.329s
2023-01-11T21:21:28.0598534Z 
2023-01-11T21:21:28.0598605Z OK (skipped=1)
2023-01-11T21:21:28.0598714Z 
2023-01-11T21:21:28.0598785Z Generating XML reports...
2023-01-11T21:21:28.0599316Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211545.xml
2023-01-11T21:21:28.0599936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0600265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0600675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0601018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0601357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplqxchvtc
2023-01-11T21:21:28.0601762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplqxchvtc/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0602224Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0602438Z 
2023-01-11T21:21:28.0602510Z Running tests...
2023-01-11T21:21:28.0602809Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0603256Z   test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2113
2023-01-11T21:21:28.0603713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2114
2023-01-11T21:21:28.0604036Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2115
2023-01-11T21:21:28.0604357Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2116
2023-01-11T21:21:28.0604789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0605113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0605532Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0605875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0606206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7b5mrbg
2023-01-11T21:21:28.0606604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7b5mrbg/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0607090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0607400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0607817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0608154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0608486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_hylz3vi
2023-01-11T21:21:28.0608868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_hylz3vi/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0609357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0609712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0610123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0610461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0610798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_p31ptam
2023-01-11T21:21:28.0611350Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_p31ptam/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0611821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0612145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0612568Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0612965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0613296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88nrzjdh
2023-01-11T21:21:28.0613687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88nrzjdh/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0614057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0614391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0614729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0615071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0615349Z skip: Need at least 1 CUDA device (2.229s)
2023-01-11T21:21:28.0615470Z 
2023-01-11T21:21:28.0615673Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0615917Z Ran 1 test in 2.229s
2023-01-11T21:21:28.0616027Z 
2023-01-11T21:21:28.0616099Z OK (skipped=1)
2023-01-11T21:21:28.0616206Z 
2023-01-11T21:21:28.0616278Z Generating XML reports...
2023-01-11T21:21:28.0616811Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211549.xml
2023-01-11T21:21:28.0617405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0617732Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0618147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0618481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0618818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj3lfg8xd
2023-01-11T21:21:28.0619206Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj3lfg8xd/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0619680Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0619894Z 
2023-01-11T21:21:28.0619966Z Running tests...
2023-01-11T21:21:28.0620262Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0620710Z   test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2168
2023-01-11T21:21:28.0621160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2169
2023-01-11T21:21:28.0621482Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2170
2023-01-11T21:21:28.0621876Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2171
2023-01-11T21:21:28.0622315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0622692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0623118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0623447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0623783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpte3f3ov_
2023-01-11T21:21:28.0624185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpte3f3ov_/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0624670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0624983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0625437Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0625781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0626125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj40zle1_
2023-01-11T21:21:28.0626508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj40zle1_/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0626991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0627311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0627722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0628063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0628402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjfvswo_7
2023-01-11T21:21:28.0628805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjfvswo_7/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0629272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0629591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0630010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0630335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0630668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdvj011_9
2023-01-11T21:21:28.0631060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdvj011_9/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0631431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0631763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0632104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0632441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0632719Z skip: Need at least 1 CUDA device (2.229s)
2023-01-11T21:21:28.0632839Z 
2023-01-11T21:21:28.0633036Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0633273Z Ran 1 test in 2.229s
2023-01-11T21:21:28.0633382Z 
2023-01-11T21:21:28.0633452Z OK (skipped=1)
2023-01-11T21:21:28.0633556Z 
2023-01-11T21:21:28.0633626Z Generating XML reports...
2023-01-11T21:21:28.0634155Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211554.xml
2023-01-11T21:21:28.0634747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0635110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0635521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0635861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0636201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp81lyws_h
2023-01-11T21:21:28.0636585Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp81lyws_h/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0637063Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0637279Z 
2023-01-11T21:21:28.0637351Z Running tests...
2023-01-11T21:21:28.0637648Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0638115Z   test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2223
2023-01-11T21:21:28.0638562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2224
2023-01-11T21:21:28.0638884Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2225
2023-01-11T21:21:28.0639204Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2226
2023-01-11T21:21:28.0639637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0639960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0640381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0640710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0641053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoysaagbm
2023-01-11T21:21:28.0641451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoysaagbm/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0641932Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0642041Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0642318Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0642451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0642631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5j4ofusd
2023-01-11T21:21:28.0642824Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5j4ofusd/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0643092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0643216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0643493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0643629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0643798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5y4ra4j8
2023-01-11T21:21:28.0643990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5y4ra4j8/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0644256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0644378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0644657Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0644852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0645035Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1wdvukx7
2023-01-11T21:21:28.0645231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1wdvukx7/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0645385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0645549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0645712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0645874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0645978Z skip: Need at least 1 CUDA device (2.332s)
2023-01-11T21:21:28.0645983Z 
2023-01-11T21:21:28.0646182Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0646291Z Ran 1 test in 2.332s
2023-01-11T21:21:28.0646297Z 
2023-01-11T21:21:28.0646373Z OK (skipped=1)
2023-01-11T21:21:28.0646377Z 
2023-01-11T21:21:28.0646464Z Generating XML reports...
2023-01-11T21:21:28.0646873Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211558.xml
2023-01-11T21:21:28.0647151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0647277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0647557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0647695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0647881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyxomjyd3
2023-01-11T21:21:28.0648084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyxomjyd3/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0648356Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0648362Z 
2023-01-11T21:21:28.0648437Z Running tests...
2023-01-11T21:21:28.0648621Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0648897Z   test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2278
2023-01-11T21:21:28.0649055Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2279
2023-01-11T21:21:28.0649209Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2280
2023-01-11T21:21:28.0649361Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2281
2023-01-11T21:21:28.0649632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0649758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0650042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0650165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0650348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_dohzj0w
2023-01-11T21:21:28.0650544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_dohzj0w/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0650940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0651079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0651364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0651555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0651739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp565bofr1
2023-01-11T21:21:28.0651936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp565bofr1/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0652192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0652314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0652593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0652728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0652912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz9qwa8zj
2023-01-11T21:21:28.0653147Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz9qwa8zj/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0653421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0653547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0653813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0653948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0654131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3yj6mq2
2023-01-11T21:21:28.0654323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3yj6mq2/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0654485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0654649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0654815Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0654976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0655077Z skip: Need at least 1 CUDA device (2.240s)
2023-01-11T21:21:28.0655083Z 
2023-01-11T21:21:28.0655267Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0655342Z Ran 1 test in 2.241s
2023-01-11T21:21:28.0655348Z 
2023-01-11T21:21:28.0655417Z OK (skipped=1)
2023-01-11T21:21:28.0655421Z 
2023-01-11T21:21:28.0655506Z Generating XML reports...
2023-01-11T21:21:28.0655923Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211602.xml
2023-01-11T21:21:28.0656194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0656320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0656595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0656730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0656901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp66j9ivs0
2023-01-11T21:21:28.0657096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp66j9ivs0/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0657362Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0657367Z 
2023-01-11T21:21:28.0657440Z Running tests...
2023-01-11T21:21:28.0657636Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0657907Z   test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2333
2023-01-11T21:21:28.0658091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2334
2023-01-11T21:21:28.0658238Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2335
2023-01-11T21:21:28.0658375Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2336
2023-01-11T21:21:28.0658649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0658769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0659047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0659178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0659364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxvo3uy_u
2023-01-11T21:21:28.0659589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxvo3uy_u/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0659859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0659979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0660249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0660380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0660561Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7mlv1vmd
2023-01-11T21:21:28.0660753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7mlv1vmd/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0661019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0661139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0661414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0661547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0661813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdgkytds0
2023-01-11T21:21:28.0661996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdgkytds0/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0662264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0662390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0662666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0662799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0662983Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_0or2cng
2023-01-11T21:21:28.0663175Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_0or2cng/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0663337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0663484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0663644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0663803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0663904Z skip: Need at least 1 CUDA device (2.240s)
2023-01-11T21:21:28.0663909Z 
2023-01-11T21:21:28.0664106Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0664186Z Ran 1 test in 2.240s
2023-01-11T21:21:28.0664191Z 
2023-01-11T21:21:28.0664301Z OK (skipped=1)
2023-01-11T21:21:28.0664306Z 
2023-01-11T21:21:28.0664391Z Generating XML reports...
2023-01-11T21:21:28.0664809Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211606.xml
2023-01-11T21:21:28.0665071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0665190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0665468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0665600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0665783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5jptn_ky
2023-01-11T21:21:28.0665974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5jptn_ky/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0666272Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0666279Z 
2023-01-11T21:21:28.0666349Z Running tests...
2023-01-11T21:21:28.0666530Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0666796Z   test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2388
2023-01-11T21:21:28.0666950Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2389
2023-01-11T21:21:28.0667099Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2390
2023-01-11T21:21:28.0667247Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2391
2023-01-11T21:21:28.0667518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0667645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0667924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0668057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0668229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgs6uy9iy
2023-01-11T21:21:28.0668427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgs6uy9iy/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0668697Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0668821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0669096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0669230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0669415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb98o1tai
2023-01-11T21:21:28.0669606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb98o1tai/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0669874Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0669982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0670253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0670383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0670565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpke0zlwwn
2023-01-11T21:21:28.0670756Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpke0zlwwn/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0671058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0671176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0671455Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0671576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0671758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9d3tiv2
2023-01-11T21:21:28.0671947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9d3tiv2/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0672107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0672270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0672454Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0672615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0672714Z skip: Need at least 1 CUDA device (2.229s)
2023-01-11T21:21:28.0672720Z 
2023-01-11T21:21:28.0672918Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0672979Z Ran 1 test in 2.229s
2023-01-11T21:21:28.0672984Z 
2023-01-11T21:21:28.0673054Z OK (skipped=1)
2023-01-11T21:21:28.0673058Z 
2023-01-11T21:21:28.0673140Z Generating XML reports...
2023-01-11T21:21:28.0673554Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211610.xml
2023-01-11T21:21:28.0673824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0673944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0674228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0674359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0674530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_niminx4
2023-01-11T21:21:28.0674720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_niminx4/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0674983Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0674988Z 
2023-01-11T21:21:28.0675058Z Running tests...
2023-01-11T21:21:28.0675252Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0675519Z   test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2443
2023-01-11T21:21:28.0675677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2444
2023-01-11T21:21:28.0675828Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2445
2023-01-11T21:21:28.0675976Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2446
2023-01-11T21:21:28.0676235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0676354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0676634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0676767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0676952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg36_qi99
2023-01-11T21:21:28.0677142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg36_qi99/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0677444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0677565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0677841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0677963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0678144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_4x_adux
2023-01-11T21:21:28.0678334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_4x_adux/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0678603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0678723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0679025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0679161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0679343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu3ujhzgz
2023-01-11T21:21:28.0679523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu3ujhzgz/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0679788Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0679909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0680186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0680318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0680502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd9n226ef
2023-01-11T21:21:28.0680694Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd9n226ef/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0680854Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0681014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0681161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0681320Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0681418Z skip: Need at least 1 CUDA device (2.328s)
2023-01-11T21:21:28.0681423Z 
2023-01-11T21:21:28.0681618Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0681694Z Ran 1 test in 2.329s
2023-01-11T21:21:28.0681700Z 
2023-01-11T21:21:28.0681770Z OK (skipped=1)
2023-01-11T21:21:28.0681774Z 
2023-01-11T21:21:28.0681862Z Generating XML reports...
2023-01-11T21:21:28.0682278Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211614.xml
2023-01-11T21:21:28.0682546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0682655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0682928Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0683059Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0683243Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo00pmghu
2023-01-11T21:21:28.0683435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo00pmghu/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0683701Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0683735Z 
2023-01-11T21:21:28.0683807Z Running tests...
2023-01-11T21:21:28.0684002Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0684264Z   test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2498
2023-01-11T21:21:28.0684417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2499
2023-01-11T21:21:28.0684566Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2500
2023-01-11T21:21:28.0684714Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2501
2023-01-11T21:21:28.0684981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0685103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0685412Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0685547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0685729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8_ucws33
2023-01-11T21:21:28.0685907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8_ucws33/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0686176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0686296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0686569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0686701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0686883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmowkoye8
2023-01-11T21:21:28.0687078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmowkoye8/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0687347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0687467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0687731Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0687863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0688045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptz937jpi
2023-01-11T21:21:28.0688235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptz937jpi/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0688505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0688628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0688907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0689039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0689211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp74gjb032
2023-01-11T21:21:28.0689400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp74gjb032/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0689563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0689722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0689879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0690081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0690178Z skip: Need at least 1 CUDA device (2.340s)
2023-01-11T21:21:28.0690184Z 
2023-01-11T21:21:28.0690381Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0690442Z Ran 1 test in 2.340s
2023-01-11T21:21:28.0690458Z 
2023-01-11T21:21:28.0690517Z OK (skipped=1)
2023-01-11T21:21:28.0690521Z 
2023-01-11T21:21:28.0690604Z Generating XML reports...
2023-01-11T21:21:28.0691175Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211618.xml
2023-01-11T21:21:28.0691453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0691575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0691900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0692039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0692221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprw1rqs71
2023-01-11T21:21:28.0692400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprw1rqs71/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0692667Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0692672Z 
2023-01-11T21:21:28.0692743Z Running tests...
2023-01-11T21:21:28.0692937Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0693206Z   test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2553
2023-01-11T21:21:28.0693360Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2554
2023-01-11T21:21:28.0693513Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2555
2023-01-11T21:21:28.0693663Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2556
2023-01-11T21:21:28.0693935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0694045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0694325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0694460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0694643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_qsr1n9t
2023-01-11T21:21:28.0694839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_qsr1n9t/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0695111Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0695234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0695512Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0695644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0695815Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_xcabcj
2023-01-11T21:21:28.0696009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_xcabcj/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0696277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0696396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0696673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0696846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0697027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4pqwwccw
2023-01-11T21:21:28.0697220Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4pqwwccw/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0697475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0697596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0697872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0698002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0698181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf85tzjke
2023-01-11T21:21:28.0698402Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf85tzjke/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0698565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0698727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0698886Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0699031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0699131Z skip: Need at least 1 CUDA device (2.231s)
2023-01-11T21:21:28.0699136Z 
2023-01-11T21:21:28.0699336Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0699413Z Ran 1 test in 2.231s
2023-01-11T21:21:28.0699418Z 
2023-01-11T21:21:28.0699489Z OK (skipped=1)
2023-01-11T21:21:28.0699494Z 
2023-01-11T21:21:28.0699579Z Generating XML reports...
2023-01-11T21:21:28.0699999Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211622.xml
2023-01-11T21:21:28.0700271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0700377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0700659Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0700793Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0700977Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4zfn6oiv
2023-01-11T21:21:28.0701171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4zfn6oiv/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0701437Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0701446Z 
2023-01-11T21:21:28.0701519Z Running tests...
2023-01-11T21:21:28.0701796Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0702071Z   test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2608
2023-01-11T21:21:28.0702214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2609
2023-01-11T21:21:28.0702367Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2610
2023-01-11T21:21:28.0702517Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2611
2023-01-11T21:21:28.0702793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0702916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0703198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0703371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0703556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqlv5qq8i
2023-01-11T21:21:28.0703751Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqlv5qq8i/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0704010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0704132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0704409Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0704544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0704729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp707klrfb
2023-01-11T21:21:28.0704952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp707klrfb/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0705224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0705349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0705610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0705744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0705928Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxzlkt63
2023-01-11T21:21:28.0706120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxzlkt63/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0706390Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0706518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0706795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0706930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0707112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprxq9b7u2
2023-01-11T21:21:28.0707289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprxq9b7u2/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0707452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0707610Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0707768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0707930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0708033Z skip: Need at least 2 CUDA devices (2.233s)
2023-01-11T21:21:28.0708038Z 
2023-01-11T21:21:28.0708234Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0708307Z Ran 1 test in 2.233s
2023-01-11T21:21:28.0708312Z 
2023-01-11T21:21:28.0708370Z OK (skipped=1)
2023-01-11T21:21:28.0708386Z 
2023-01-11T21:21:28.0708457Z Generating XML reports...
2023-01-11T21:21:28.0708876Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211627.xml
2023-01-11T21:21:28.0709146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0709266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0709545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0709711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0709894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdras52q7
2023-01-11T21:21:28.0710088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdras52q7/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0710343Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0710362Z 
2023-01-11T21:21:28.0710421Z Running tests...
2023-01-11T21:21:28.0710618Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0711364Z   test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79750 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.307s)
2023-01-11T21:21:28.0711372Z 
2023-01-11T21:21:28.0711569Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0711642Z Ran 1 test in 0.307s
2023-01-11T21:21:28.0711647Z 
2023-01-11T21:21:28.0711715Z OK (skipped=1)
2023-01-11T21:21:28.0711720Z 
2023-01-11T21:21:28.0711802Z Generating XML reports...
2023-01-11T21:21:28.0712215Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211631.xml
2023-01-11T21:21:28.0712488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0712600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0712875Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0713010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0713197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo928o4_f
2023-01-11T21:21:28.0713393Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo928o4_f/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0713655Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0713661Z 
2023-01-11T21:21:28.0713731Z Running tests...
2023-01-11T21:21:28.0713925Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0714187Z   test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2673
2023-01-11T21:21:28.0714327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2674
2023-01-11T21:21:28.0714476Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2675
2023-01-11T21:21:28.0714629Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2676
2023-01-11T21:21:28.0714896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0715017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0715296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0715428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0715610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3t3q1dwo
2023-01-11T21:21:28.0715792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3t3q1dwo/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0716061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0716184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0716491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0716622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0716801Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfr60x9qs
2023-01-11T21:21:28.0716990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfr60x9qs/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0717256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0717376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0717642Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0717770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0717981Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc6f0xxd4
2023-01-11T21:21:28.0718173Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc6f0xxd4/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0718444Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0718564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0718839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0718969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0719151Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzqc21r2q
2023-01-11T21:21:28.0719329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzqc21r2q/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0719495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0719658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0719817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0719975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0720078Z skip: Need at least 2 CUDA devices (2.231s)
2023-01-11T21:21:28.0720083Z 
2023-01-11T21:21:28.0720275Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0720349Z Ran 1 test in 2.231s
2023-01-11T21:21:28.0720354Z 
2023-01-11T21:21:28.0720413Z OK (skipped=1)
2023-01-11T21:21:28.0720428Z 
2023-01-11T21:21:28.0720498Z Generating XML reports...
2023-01-11T21:21:28.0720913Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211633.xml
2023-01-11T21:21:28.0721183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0721305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0721581Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0721715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0721898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprjssh6nk
2023-01-11T21:21:28.0722091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprjssh6nk/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0722345Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0722351Z 
2023-01-11T21:21:28.0722422Z Running tests...
2023-01-11T21:21:28.0722617Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0722910Z   test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2728
2023-01-11T21:21:28.0723062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2729
2023-01-11T21:21:28.0723210Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2730
2023-01-11T21:21:28.0723358Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2731
2023-01-11T21:21:28.0723629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0723749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0724016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0724175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0724361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptf5le5un
2023-01-11T21:21:28.0724552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptf5le5un/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0724822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0724944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0725222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0725353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0725525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpweowp6ul
2023-01-11T21:21:28.0725718Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpweowp6ul/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0725987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0726107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0726379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0726509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0726689Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsl2nsx0i
2023-01-11T21:21:28.0726880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsl2nsx0i/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0727147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0727256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0727534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0727668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0727848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpig0q8nll
2023-01-11T21:21:28.0728036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpig0q8nll/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0728196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0728357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0728512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0728660Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0728761Z skip: Need at least 2 CUDA devices (2.228s)
2023-01-11T21:21:28.0728798Z 
2023-01-11T21:21:28.0728994Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0729068Z Ran 1 test in 2.228s
2023-01-11T21:21:28.0729073Z 
2023-01-11T21:21:28.0729142Z OK (skipped=1)
2023-01-11T21:21:28.0729147Z 
2023-01-11T21:21:28.0729228Z Generating XML reports...
2023-01-11T21:21:28.0729642Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211637.xml
2023-01-11T21:21:28.0729914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0730034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0730298Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0730430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0730639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv4qijeq4
2023-01-11T21:21:28.0730974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv4qijeq4/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0731254Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0731260Z 
2023-01-11T21:21:28.0731331Z Running tests...
2023-01-11T21:21:28.0731527Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0731796Z   test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2783
2023-01-11T21:21:28.0731953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2784
2023-01-11T21:21:28.0732091Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2785
2023-01-11T21:21:28.0732242Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2786
2023-01-11T21:21:28.0732518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0732640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0732919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0733052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0733238Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm0ru4f0f
2023-01-11T21:21:28.0733431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm0ru4f0f/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0733689Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0733809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0734091Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0734223Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0755007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ks0u0cl
2023-01-11T21:21:28.0755298Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ks0u0cl/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0755623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0755744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0756022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0756148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0756333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4w7jegzy
2023-01-11T21:21:28.0756672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4w7jegzy/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0756939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0757055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0757323Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0757448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0757624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9nitq0_c
2023-01-11T21:21:28.0757813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9nitq0_c/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0758015Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0758173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0758327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0758478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0758571Z skip: Need at least 2 CUDA devices (2.226s)
2023-01-11T21:21:28.0758578Z 
2023-01-11T21:21:28.0758766Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0758834Z Ran 1 test in 2.226s
2023-01-11T21:21:28.0758839Z 
2023-01-11T21:21:28.0758901Z OK (skipped=1)
2023-01-11T21:21:28.0758905Z 
2023-01-11T21:21:28.0758982Z Generating XML reports...
2023-01-11T21:21:28.0759401Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211641.xml
2023-01-11T21:21:28.0759671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0759789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0760063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0760184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0760360Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps7ssofah
2023-01-11T21:21:28.0760545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps7ssofah/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0760801Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0760807Z 
2023-01-11T21:21:28.0760872Z Running tests...
2023-01-11T21:21:28.0761060Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0761315Z   test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2838
2023-01-11T21:21:28.0761460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2839
2023-01-11T21:21:28.0761604Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 2840
2023-01-11T21:21:28.0761739Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 2841
2023-01-11T21:21:28.0762003Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0762114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0762386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0762510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0762692Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2tjl72hg
2023-01-11T21:21:28.0762930Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2tjl72hg/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0763202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0763321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0763585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0763718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0763901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9065wkj7
2023-01-11T21:21:28.0764091Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9065wkj7/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0764393Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0764519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0764794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0764926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0765095Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpurlt5ync
2023-01-11T21:21:28.0765286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpurlt5ync/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0765555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0765675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0765955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0766091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0766277Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsbdm6b4e
2023-01-11T21:21:28.0766470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsbdm6b4e/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0766632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0766782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0766944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0767103Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0767171Z ok (2.527s)
2023-01-11T21:21:28.0767176Z 
2023-01-11T21:21:28.0767373Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0767453Z Ran 1 test in 2.527s
2023-01-11T21:21:28.0767459Z 
2023-01-11T21:21:28.0767519Z OK
2023-01-11T21:21:28.0767524Z 
2023-01-11T21:21:28.0767606Z Generating XML reports...
2023-01-11T21:21:28.0768014Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211645.xml
2023-01-11T21:21:28.0768288Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0768411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0768692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0768827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0769008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpon1b_y25
2023-01-11T21:21:28.0769202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpon1b_y25/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0769502Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0769507Z 
2023-01-11T21:21:28.0769579Z Running tests...
2023-01-11T21:21:28.0769760Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0770034Z   test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3061
2023-01-11T21:21:28.0770189Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3062
2023-01-11T21:21:28.0770338Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3063
2023-01-11T21:21:28.0770489Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3064
2023-01-11T21:21:28.0770790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0771062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0771349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0771484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0771657Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsub257by
2023-01-11T21:21:28.0771852Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsub257by/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0772124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0772246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0772527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0772666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0772851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppfhbo67m
2023-01-11T21:21:28.0773124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0773233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0773509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0773642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0773825Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgvtod1gc
2023-01-11T21:21:28.0774020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppfhbo67m/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0774216Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgvtod1gc/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0774485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0774608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0774885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0775005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0775186Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp34e7qmx9
2023-01-11T21:21:28.0775378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp34e7qmx9/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0775539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0775702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0775905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0776065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0776166Z skip: Need at least 1 CUDA device (2.231s)
2023-01-11T21:21:28.0776171Z 
2023-01-11T21:21:28.0776355Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0776430Z Ran 1 test in 2.231s
2023-01-11T21:21:28.0776435Z 
2023-01-11T21:21:28.0776506Z OK (skipped=1)
2023-01-11T21:21:28.0776511Z 
2023-01-11T21:21:28.0776594Z Generating XML reports...
2023-01-11T21:21:28.0777018Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211649.xml
2023-01-11T21:21:28.0777292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0777449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0777734Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0777867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0778038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1q22e0kp
2023-01-11T21:21:28.0778232Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1q22e0kp/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0778496Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0778501Z 
2023-01-11T21:21:28.0778573Z Running tests...
2023-01-11T21:21:28.0778770Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0779047Z   test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3116
2023-01-11T21:21:28.0779203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3117
2023-01-11T21:21:28.0779353Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3118
2023-01-11T21:21:28.0779504Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3119
2023-01-11T21:21:28.0779762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0779884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0780162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0780296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0780481Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_rydfgb
2023-01-11T21:21:28.0780678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_rydfgb/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0780952Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0781073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0781334Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0781467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0781737Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qf2pvxc
2023-01-11T21:21:28.0781919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qf2pvxc/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0782190Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0782317Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0782629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0782762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0782945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeuk1lmuv
2023-01-11T21:21:28.0783136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeuk1lmuv/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0783403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0783512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0783790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0783922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0784137Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpui2cfed6
2023-01-11T21:21:28.0784332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpui2cfed6/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0784494Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0784654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0784816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0784975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0785065Z skip: Need at least 2 CUDA devices (2.228s)
2023-01-11T21:21:28.0785072Z 
2023-01-11T21:21:28.0785268Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0785343Z Ran 1 test in 2.229s
2023-01-11T21:21:28.0785347Z 
2023-01-11T21:21:28.0785424Z OK (skipped=1)
2023-01-11T21:21:28.0785429Z 
2023-01-11T21:21:28.0785515Z Generating XML reports...
2023-01-11T21:21:28.0785935Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211653.xml
2023-01-11T21:21:28.0786207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0786328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0786603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0786724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0786905Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpam_w72ui
2023-01-11T21:21:28.0787098Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpam_w72ui/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0787369Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0787375Z 
2023-01-11T21:21:28.0787446Z Running tests...
2023-01-11T21:21:28.0787642Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0787907Z   test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3171
2023-01-11T21:21:28.0788063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3172
2023-01-11T21:21:28.0788202Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3173
2023-01-11T21:21:28.0788355Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3174
2023-01-11T21:21:28.0788623Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0788778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0789058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0789193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0789377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxkohqz33
2023-01-11T21:21:28.0789571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxkohqz33/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0789844Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0789952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0790228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0790390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0790574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb_lgdboi
2023-01-11T21:21:28.0790845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0790965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0791244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0791375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0791544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp14kwy452
2023-01-11T21:21:28.0791736Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb_lgdboi/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0791922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp14kwy452/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0792194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0792314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0792590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0792723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0792908Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp43d1kl1r
2023-01-11T21:21:28.0793094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp43d1kl1r/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0793243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0793405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0793567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0793726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0793828Z skip: Need at least 2 CUDA devices (2.230s)
2023-01-11T21:21:28.0793834Z 
2023-01-11T21:21:28.0794031Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0794108Z Ran 1 test in 2.231s
2023-01-11T21:21:28.0794113Z 
2023-01-11T21:21:28.0794182Z OK (skipped=1)
2023-01-11T21:21:28.0794187Z 
2023-01-11T21:21:28.0794271Z Generating XML reports...
2023-01-11T21:21:28.0794674Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211657.xml
2023-01-11T21:21:28.0794947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0795069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0795379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0795516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0795699Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph1lyic4k
2023-01-11T21:21:28.0795893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph1lyic4k/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0796159Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0796164Z 
2023-01-11T21:21:28.0796235Z Running tests...
2023-01-11T21:21:28.0796417Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0797210Z   test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/80008 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.309s)
2023-01-11T21:21:28.0797218Z 
2023-01-11T21:21:28.0797408Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0797482Z Ran 1 test in 0.310s
2023-01-11T21:21:28.0797487Z 
2023-01-11T21:21:28.0797558Z OK (skipped=1)
2023-01-11T21:21:28.0797562Z 
2023-01-11T21:21:28.0797647Z Generating XML reports...
2023-01-11T21:21:28.0798061Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211701.xml
2023-01-11T21:21:28.0798332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0798456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0798737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0798860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0799043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl_5zpfdy
2023-01-11T21:21:28.0799234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl_5zpfdy/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0799497Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0799502Z 
2023-01-11T21:21:28.0799576Z Running tests...
2023-01-11T21:21:28.0799769Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0800034Z   test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3236
2023-01-11T21:21:28.0800190Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3237
2023-01-11T21:21:28.0800331Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3238
2023-01-11T21:21:28.0800481Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3239
2023-01-11T21:21:28.0800754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0800877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0801157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0801294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0801478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3rrfti9q
2023-01-11T21:21:28.0801672Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3rrfti9q/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0801987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0802097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0802375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0802506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0802686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpif1gwj3r
2023-01-11T21:21:28.0802879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpif1gwj3r/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0803149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0803270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0803575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0803698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0803880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpubojgxo6
2023-01-11T21:21:28.0804075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpubojgxo6/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0804343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0804463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0804739Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0804874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0805058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxc39zl6u
2023-01-11T21:21:28.0805256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxc39zl6u/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0805407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0805566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0805727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0805888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0805989Z skip: Need at least 2 CUDA devices (2.235s)
2023-01-11T21:21:28.0805995Z 
2023-01-11T21:21:28.0806194Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0806269Z Ran 1 test in 2.236s
2023-01-11T21:21:28.0806275Z 
2023-01-11T21:21:28.0806346Z OK (skipped=1)
2023-01-11T21:21:28.0806351Z 
2023-01-11T21:21:28.0806422Z Generating XML reports...
2023-01-11T21:21:28.0806842Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211704.xml
2023-01-11T21:21:28.0807110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0807233Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0807511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0807644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0807827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv8e6dkot
2023-01-11T21:21:28.0808021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv8e6dkot/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0808292Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0808326Z 
2023-01-11T21:21:28.0808386Z Running tests...
2023-01-11T21:21:28.0808584Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0808850Z   test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3291
2023-01-11T21:21:28.0809008Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3292
2023-01-11T21:21:28.0809160Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3293
2023-01-11T21:21:28.0809311Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3294
2023-01-11T21:21:28.0809587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0809712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0810020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0810144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0810332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiym3fahc
2023-01-11T21:21:28.0810526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiym3fahc/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0810798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0811077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0811365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0811499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0811688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp446ypd_n
2023-01-11T21:21:28.0811873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp446ypd_n/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0812147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0812269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0812547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0812683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0812866Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgy30kg7b
2023-01-11T21:21:28.0813060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgy30kg7b/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0813336Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0813461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0813727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0813861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0814043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjycf_7tj
2023-01-11T21:21:28.0814238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjycf_7tj/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0814400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0814564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0814722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0814884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0815031Z skip: Need at least 2 CUDA devices (2.344s)
2023-01-11T21:21:28.0815049Z 
2023-01-11T21:21:28.0815235Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0815311Z Ran 1 test in 2.344s
2023-01-11T21:21:28.0815316Z 
2023-01-11T21:21:28.0815391Z OK (skipped=1)
2023-01-11T21:21:28.0815395Z 
2023-01-11T21:21:28.0815481Z Generating XML reports...
2023-01-11T21:21:28.0815901Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211708.xml
2023-01-11T21:21:28.0816174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0816298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0816615Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0816739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0816923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjagldmb9
2023-01-11T21:21:28.0817118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjagldmb9/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0817385Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0817390Z 
2023-01-11T21:21:28.0817463Z Running tests...
2023-01-11T21:21:28.0817658Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0817923Z   test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3346
2023-01-11T21:21:28.0818078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3347
2023-01-11T21:21:28.0818230Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3348
2023-01-11T21:21:28.0818371Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3349
2023-01-11T21:21:28.0818643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0818765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0819047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0819183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0819368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp57sy1r2k
2023-01-11T21:21:28.0819565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp57sy1r2k/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0819841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0819966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0820228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0820361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0820541Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97y_yjro
2023-01-11T21:21:28.0820729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97y_yjro/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0820991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0821106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0821377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0821539Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0821770Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgjm5_k3a
2023-01-11T21:21:28.0821957Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgjm5_k3a/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0822222Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0822334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0822603Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0822736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0822917Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_paf_oz
2023-01-11T21:21:28.0823140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_paf_oz/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0823307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0823457Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0823619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0823773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0823876Z skip: Need at least 2 CUDA devices (2.341s)
2023-01-11T21:21:28.0823881Z 
2023-01-11T21:21:28.0824077Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0824151Z Ran 1 test in 2.342s
2023-01-11T21:21:28.0824157Z 
2023-01-11T21:21:28.0824229Z OK (skipped=1)
2023-01-11T21:21:28.0824233Z 
2023-01-11T21:21:28.0824317Z Generating XML reports...
2023-01-11T21:21:28.0824726Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211712.xml
2023-01-11T21:21:28.0825000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0825122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0825398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0825529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0825714Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdlo8kwlj
2023-01-11T21:21:28.0825907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdlo8kwlj/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0826174Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0826179Z 
2023-01-11T21:21:28.0826254Z Running tests...
2023-01-11T21:21:28.0826436Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0826700Z   test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3401
2023-01-11T21:21:28.0826854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3402
2023-01-11T21:21:28.0827004Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3403
2023-01-11T21:21:28.0827153Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3404
2023-01-11T21:21:28.0827423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0827544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0827820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0827986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0828157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1wj2s5qy
2023-01-11T21:21:28.0828427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0828548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0828825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0828957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0829140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp48nqred2
2023-01-11T21:21:28.0829332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1wj2s5qy/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0829549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp48nqred2/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0829812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0829932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0830207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0830339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0830519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpknwxsvjb
2023-01-11T21:21:28.0830712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpknwxsvjb/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0830977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0831103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0831385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0831506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0831688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpddrwqx0p
2023-01-11T21:21:28.0831883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpddrwqx0p/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0832046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0832206Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0832364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0832521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0832627Z skip: Need at least 2 CUDA devices (2.336s)
2023-01-11T21:21:28.0832632Z 
2023-01-11T21:21:28.0832814Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0832886Z Ran 1 test in 2.336s
2023-01-11T21:21:28.0832891Z 
2023-01-11T21:21:28.0832962Z OK (skipped=1)
2023-01-11T21:21:28.0832967Z 
2023-01-11T21:21:28.0833050Z Generating XML reports...
2023-01-11T21:21:28.0833466Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211716.xml
2023-01-11T21:21:28.0833737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0833859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0834134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0834268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0834468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_n96sq7
2023-01-11T21:21:28.0834662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_n96sq7/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0834929Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0834934Z 
2023-01-11T21:21:28.0835006Z Running tests...
2023-01-11T21:21:28.0835200Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0835462Z   test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3456
2023-01-11T21:21:28.0835616Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3457
2023-01-11T21:21:28.0835765Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3458
2023-01-11T21:21:28.0835944Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3459
2023-01-11T21:21:28.0836204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0836326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0836604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0836737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0836920Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy889f5b7
2023-01-11T21:21:28.0837112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy889f5b7/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0837383Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0837505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0837771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0837904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0838087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiikql5tk
2023-01-11T21:21:28.0838280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiikql5tk/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0838547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0838669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0838946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0839077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0839265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd5cdhv_m
2023-01-11T21:21:28.0839448Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd5cdhv_m/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0839713Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0839834Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0840113Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0840246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0840428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0fat7e52
2023-01-11T21:21:28.0840620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0fat7e52/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0840817Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0840968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0841126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0841285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0841389Z skip: Need at least 2 CUDA devices (2.234s)
2023-01-11T21:21:28.0841395Z 
2023-01-11T21:21:28.0841591Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0841665Z Ran 1 test in 2.235s
2023-01-11T21:21:28.0841670Z 
2023-01-11T21:21:28.0841741Z OK (skipped=1)
2023-01-11T21:21:28.0841746Z 
2023-01-11T21:21:28.0841829Z Generating XML reports...
2023-01-11T21:21:28.0842279Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211720.xml
2023-01-11T21:21:28.0842544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0842665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0842943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0843075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0843263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpobre5p_6
2023-01-11T21:21:28.0843459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpobre5p_6/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0843723Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0843728Z 
2023-01-11T21:21:28.0843801Z Running tests...
2023-01-11T21:21:28.0843995Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0844248Z   test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3511
2023-01-11T21:21:28.0844400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3512
2023-01-11T21:21:28.0844549Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3513
2023-01-11T21:21:28.0844698Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3514
2023-01-11T21:21:28.0844968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0845089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0845366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0845502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0845674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfsdm327l
2023-01-11T21:21:28.0845870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfsdm327l/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0846137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0846258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0846538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0846670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0846853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkpkuccld
2023-01-11T21:21:28.0847047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkpkuccld/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0847358Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0847467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0847745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0847877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0848060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqexp2ykp
2023-01-11T21:21:28.0848255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqexp2ykp/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0848523Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0848644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0848954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0849089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0849258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf9za_c7q
2023-01-11T21:21:28.0849450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf9za_c7q/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0849612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0849774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0849934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0850097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0850200Z skip: Need at least 2 CUDA devices (2.331s)
2023-01-11T21:21:28.0850207Z 
2023-01-11T21:21:28.0850407Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0850470Z Ran 1 test in 2.331s
2023-01-11T21:21:28.0850475Z 
2023-01-11T21:21:28.0850544Z OK (skipped=1)
2023-01-11T21:21:28.0850549Z 
2023-01-11T21:21:28.0850635Z Generating XML reports...
2023-01-11T21:21:28.0851185Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211724.xml
2023-01-11T21:21:28.0851465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0851588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0851868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0852003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0852191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qeo75kq
2023-01-11T21:21:28.0852370Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qeo75kq/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0852636Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0852641Z 
2023-01-11T21:21:28.0852713Z Running tests...
2023-01-11T21:21:28.0852909Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0853172Z   test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3566
2023-01-11T21:21:28.0853329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3567
2023-01-11T21:21:28.0853480Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3568
2023-01-11T21:21:28.0853632Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3569
2023-01-11T21:21:28.0853953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0854063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0854342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0854474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0854662Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcogx83o6
2023-01-11T21:21:28.0854857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcogx83o6/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0855126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0855248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0855560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0855682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0855867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvbc6s1hc
2023-01-11T21:21:28.0856061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvbc6s1hc/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0856331Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0856451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0856726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0856858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0857043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp00xpa01q
2023-01-11T21:21:28.0857236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp00xpa01q/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0857491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0857613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0857890Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0858022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0858203Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5c5vkrvp
2023-01-11T21:21:28.0858397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5c5vkrvp/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0858559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0858727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0858873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0859031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0859134Z skip: Need at least 2 CUDA devices (2.328s)
2023-01-11T21:21:28.0859139Z 
2023-01-11T21:21:28.0859332Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0859406Z Ran 1 test in 2.328s
2023-01-11T21:21:28.0859411Z 
2023-01-11T21:21:28.0859480Z OK (skipped=1)
2023-01-11T21:21:28.0859484Z 
2023-01-11T21:21:28.0859567Z Generating XML reports...
2023-01-11T21:21:28.0859984Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211728.xml
2023-01-11T21:21:28.0860257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0860396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0860672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0860805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0860990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiv477zpg
2023-01-11T21:21:28.0861182Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiv477zpg/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0861446Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0861452Z 
2023-01-11T21:21:28.0861522Z Running tests...
2023-01-11T21:21:28.0861779Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0862080Z   test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3621
2023-01-11T21:21:28.0862223Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3622
2023-01-11T21:21:28.0862374Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3623
2023-01-11T21:21:28.0862525Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3624
2023-01-11T21:21:28.0862798Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0862922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0863204Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0863339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0863528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb7a30f1y
2023-01-11T21:21:28.0863712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb7a30f1y/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0863987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0864110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0864387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0864522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0864707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpese7adrg
2023-01-11T21:21:28.0864902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpese7adrg/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0865172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0865297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0865564Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0865702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0865888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi03aat2w
2023-01-11T21:21:28.0866083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi03aat2w/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0866352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0866474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0866751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0866920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0867090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptwsncskf
2023-01-11T21:21:28.0867288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptwsncskf/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0867451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0867616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0867777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0867936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0868039Z skip: Need at least 2 CUDA devices (2.228s)
2023-01-11T21:21:28.0868044Z 
2023-01-11T21:21:28.0868302Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0868382Z Ran 1 test in 2.229s
2023-01-11T21:21:28.0868387Z 
2023-01-11T21:21:28.0868446Z OK (skipped=1)
2023-01-11T21:21:28.0868451Z 
2023-01-11T21:21:28.0868537Z Generating XML reports...
2023-01-11T21:21:28.0868960Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211733.xml
2023-01-11T21:21:28.0869235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0869360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0869641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0869774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0869957Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3i1u7lzw
2023-01-11T21:21:28.0870156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3i1u7lzw/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0870412Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0870417Z 
2023-01-11T21:21:28.0870491Z Running tests...
2023-01-11T21:21:28.0870686Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0870958Z   test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3676
2023-01-11T21:21:28.0871115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3677
2023-01-11T21:21:28.0871265Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3678
2023-01-11T21:21:28.0871415Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3679
2023-01-11T21:21:28.0871692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0871804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0872085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0872219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0872405Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpupkt36si
2023-01-11T21:21:28.0872599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpupkt36si/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0872869Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0872992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0873270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0873438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0873608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsu16bn02
2023-01-11T21:21:28.0873801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsu16bn02/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0874070Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0874190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0874469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0874603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0874784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppde8ssge
2023-01-11T21:21:28.0875009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppde8ssge/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0875265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0875389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0875666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0875797Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0875978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8o1kjr9h
2023-01-11T21:21:28.0876170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8o1kjr9h/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0876332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0876495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0876655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0876803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0876904Z skip: Need at least 2 CUDA devices (2.227s)
2023-01-11T21:21:28.0876909Z 
2023-01-11T21:21:28.0877105Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0877179Z Ran 1 test in 2.227s
2023-01-11T21:21:28.0877184Z 
2023-01-11T21:21:28.0877253Z OK (skipped=1)
2023-01-11T21:21:28.0877257Z 
2023-01-11T21:21:28.0877339Z Generating XML reports...
2023-01-11T21:21:28.0877757Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211737.xml
2023-01-11T21:21:28.0878032Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0878156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0878423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0878557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0878739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp11q3dz22
2023-01-11T21:21:28.0878932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp11q3dz22/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0879199Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0879204Z 
2023-01-11T21:21:28.0879275Z Running tests...
2023-01-11T21:21:28.0879469Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0879742Z   test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3731
2023-01-11T21:21:28.0879925Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3732
2023-01-11T21:21:28.0880062Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3733
2023-01-11T21:21:28.0880213Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3734
2023-01-11T21:21:28.0880483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0880603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0880883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0881016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0881201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp22wo835
2023-01-11T21:21:28.0881423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp22wo835/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0881686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0881806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0882083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0882216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0882399Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd9ywkzvu
2023-01-11T21:21:28.0882594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd9ywkzvu/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0882862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0882987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0883263Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0883383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0883564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5m8cjs6i
2023-01-11T21:21:28.0883757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5m8cjs6i/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0884023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0884144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0884422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0884556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0884738Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1a39f_6h
2023-01-11T21:21:28.0884916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1a39f_6h/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0885078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0885241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0885400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0885558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0885659Z skip: Need at least 2 CUDA devices (2.325s)
2023-01-11T21:21:28.0885664Z 
2023-01-11T21:21:28.0885860Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0885935Z Ran 1 test in 2.325s
2023-01-11T21:21:28.0885973Z 
2023-01-11T21:21:28.0886045Z OK (skipped=1)
2023-01-11T21:21:28.0886050Z 
2023-01-11T21:21:28.0886120Z Generating XML reports...
2023-01-11T21:21:28.0886535Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211741.xml
2023-01-11T21:21:28.0886807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0886927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0887206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0887339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0887522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnvjtf5n3
2023-01-11T21:21:28.0887742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnvjtf5n3/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0888000Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0888018Z 
2023-01-11T21:21:28.0888077Z Running tests...
2023-01-11T21:21:28.0888271Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0888536Z   test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3786
2023-01-11T21:21:28.0888692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3787
2023-01-11T21:21:28.0888841Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3788
2023-01-11T21:21:28.0888988Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3789
2023-01-11T21:21:28.0889266Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0889391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0889656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0889790Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0889975Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxmyghjoj
2023-01-11T21:21:28.0890168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxmyghjoj/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0890435Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0890555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0890959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0891099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0891281Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7sr6_3dh
2023-01-11T21:21:28.0891461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7sr6_3dh/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0891735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0891856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0892134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0892266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0892450Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl8rx376o
2023-01-11T21:21:28.0892646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl8rx376o/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0892974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0893082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0893362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0893497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0893681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3dkxjae3
2023-01-11T21:21:28.0893873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3dkxjae3/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0894037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0894200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0894396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0894559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0894649Z skip: Need at least 2 CUDA devices (2.224s)
2023-01-11T21:21:28.0894655Z 
2023-01-11T21:21:28.0894854Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0894929Z Ran 1 test in 2.224s
2023-01-11T21:21:28.0894934Z 
2023-01-11T21:21:28.0895006Z OK (skipped=1)
2023-01-11T21:21:28.0895011Z 
2023-01-11T21:21:28.0895094Z Generating XML reports...
2023-01-11T21:21:28.0895512Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211745.xml
2023-01-11T21:21:28.0895782Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0895904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0896172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0896306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0896490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpum67qw68
2023-01-11T21:21:28.0896685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpum67qw68/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0896952Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0896957Z 
2023-01-11T21:21:28.0897028Z Running tests...
2023-01-11T21:21:28.0897222Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0897494Z   test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3841
2023-01-11T21:21:28.0897650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3842
2023-01-11T21:21:28.0897787Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3843
2023-01-11T21:21:28.0897939Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3844
2023-01-11T21:21:28.0898212Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0898333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0898610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0898743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0898931Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi2vvdiu_
2023-01-11T21:21:28.0899127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi2vvdiu_/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0899426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0899535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0899809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0899941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0900123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjipcin3v
2023-01-11T21:21:28.0900316Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjipcin3v/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0900584Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0900736Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0901015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0901134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0901316Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqtbntizw
2023-01-11T21:21:28.0901509Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqtbntizw/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0901852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0901975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0902252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0902386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0902571Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphs6nok1f
2023-01-11T21:21:28.0902764Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphs6nok1f/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0902915Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0903078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0903239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0903397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0903499Z skip: Need at least 2 CUDA devices (2.228s)
2023-01-11T21:21:28.0903505Z 
2023-01-11T21:21:28.0903702Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0903777Z Ran 1 test in 2.228s
2023-01-11T21:21:28.0903782Z 
2023-01-11T21:21:28.0903858Z OK (skipped=1)
2023-01-11T21:21:28.0903863Z 
2023-01-11T21:21:28.0903934Z Generating XML reports...
2023-01-11T21:21:28.0904352Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211749.xml
2023-01-11T21:21:28.0904626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0904748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0905023Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0905156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0905337Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph5agf6i2
2023-01-11T21:21:28.0905527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph5agf6i2/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0905829Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0905835Z 
2023-01-11T21:21:28.0905895Z Running tests...
2023-01-11T21:21:28.0906089Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0906356Z   test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3896
2023-01-11T21:21:28.0906511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3897
2023-01-11T21:21:28.0906661Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3898
2023-01-11T21:21:28.0906809Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3899
2023-01-11T21:21:28.0907082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0907229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0907516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0907637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0907822Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb90xl04m
2023-01-11T21:21:28.0908015Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb90xl04m/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0908282Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0908402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0908678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0908810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0908999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmdybnyz
2023-01-11T21:21:28.0909181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmdybnyz/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0909448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0909568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0909846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0909976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0910157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgwgkgj5
2023-01-11T21:21:28.0910351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgwgkgj5/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0910625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0910747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0911010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0911142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0911323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4wwrh_7k
2023-01-11T21:21:28.0911513Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4wwrh_7k/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0911673Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0911835Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0911995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0912183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0912273Z skip: Need at least 2 CUDA devices (2.331s)
2023-01-11T21:21:28.0912290Z 
2023-01-11T21:21:28.0912472Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0912549Z Ran 1 test in 2.331s
2023-01-11T21:21:28.0912553Z 
2023-01-11T21:21:28.0912624Z OK (skipped=1)
2023-01-11T21:21:28.0912628Z 
2023-01-11T21:21:28.0912714Z Generating XML reports...
2023-01-11T21:21:28.0913132Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211753.xml
2023-01-11T21:21:28.0913404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0913524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0913835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0913956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0914138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfmu5r2hb
2023-01-11T21:21:28.0914331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfmu5r2hb/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0914599Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0914605Z 
2023-01-11T21:21:28.0914675Z Running tests...
2023-01-11T21:21:28.0914868Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0915139Z   test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3951
2023-01-11T21:21:28.0915295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3952
2023-01-11T21:21:28.0915446Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 3953
2023-01-11T21:21:28.0915583Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 3954
2023-01-11T21:21:28.0915857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0915978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0916255Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0916392Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0916575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5_78ydvx
2023-01-11T21:21:28.0916768Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5_78ydvx/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0917040Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0917160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0917421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0917554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0917739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuklxrp5r
2023-01-11T21:21:28.0917935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuklxrp5r/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0918206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0918328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0918609Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0918773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0918944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp3hq58fc
2023-01-11T21:21:28.0919138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp3hq58fc/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0919406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0919528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0919807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0919941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0920159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplz6o4xdg
2023-01-11T21:21:28.0920357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplz6o4xdg/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0920523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0920674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0920837Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0920996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0921099Z skip: Need at least 2 CUDA devices (2.332s)
2023-01-11T21:21:28.0921105Z 
2023-01-11T21:21:28.0921301Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0921379Z Ran 1 test in 2.333s
2023-01-11T21:21:28.0921384Z 
2023-01-11T21:21:28.0921456Z OK (skipped=1)
2023-01-11T21:21:28.0921461Z 
2023-01-11T21:21:28.0921550Z Generating XML reports...
2023-01-11T21:21:28.0921953Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211757.xml
2023-01-11T21:21:28.0922226Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0922350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0922629Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0922760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0922945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu89odbfu
2023-01-11T21:21:28.0923141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu89odbfu/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0923408Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0923415Z 
2023-01-11T21:21:28.0923489Z Running tests...
2023-01-11T21:21:28.0923670Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0923936Z   test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4006
2023-01-11T21:21:28.0924091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4007
2023-01-11T21:21:28.0924243Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4008
2023-01-11T21:21:28.0924394Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4009
2023-01-11T21:21:28.0924666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0924790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0925104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0925238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0925411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdjsg00ir
2023-01-11T21:21:28.0925607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdjsg00ir/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0925878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0926000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0926277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0926411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0926621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpie_lyc05
2023-01-11T21:21:28.0926820Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpie_lyc05/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0927078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0927200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0927480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0927613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0927795Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj_da5azb
2023-01-11T21:21:28.0927987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj_da5azb/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0928259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0928384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0928663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0928784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0928969Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp87sy6gw4
2023-01-11T21:21:28.0929159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp87sy6gw4/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0929319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0929479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0929639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0929805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0929909Z skip: Need at least 2 CUDA devices (2.329s)
2023-01-11T21:21:28.0929914Z 
2023-01-11T21:21:28.0930097Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0930171Z Ran 1 test in 2.329s
2023-01-11T21:21:28.0930176Z 
2023-01-11T21:21:28.0930245Z OK (skipped=1)
2023-01-11T21:21:28.0930250Z 
2023-01-11T21:21:28.0930334Z Generating XML reports...
2023-01-11T21:21:28.0930752Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211801.xml
2023-01-11T21:21:28.0931146Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0931269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0931550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0931734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0931909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo6g48wi8
2023-01-11T21:21:28.0932102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo6g48wi8/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0932371Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0932376Z 
2023-01-11T21:21:28.0932449Z Running tests...
2023-01-11T21:21:28.0932646Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0932916Z   test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4061
2023-01-11T21:21:28.0933071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4062
2023-01-11T21:21:28.0933255Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4063
2023-01-11T21:21:28.0933407Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4064
2023-01-11T21:21:28.0933668Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0933792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0934071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0934204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0934388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgp1n6m5
2023-01-11T21:21:28.0934584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgp1n6m5/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0934857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0934979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0935240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0935372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0935552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_pqhsvpn
2023-01-11T21:21:28.0935743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_pqhsvpn/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0936009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0936128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0936407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0936545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0936727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdms7x34v
2023-01-11T21:21:28.0936907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdms7x34v/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0937173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0937294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0937569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0937700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0937881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpatala1jf
2023-01-11T21:21:28.0938161Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpatala1jf/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0938323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0938471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0938631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0938790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0938894Z skip: Need at least 2 CUDA devices (2.332s)
2023-01-11T21:21:28.0938900Z 
2023-01-11T21:21:28.0939095Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0939170Z Ran 1 test in 2.332s
2023-01-11T21:21:28.0939175Z 
2023-01-11T21:21:28.0939251Z OK (skipped=1)
2023-01-11T21:21:28.0939255Z 
2023-01-11T21:21:28.0939341Z Generating XML reports...
2023-01-11T21:21:28.0939792Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211805.xml
2023-01-11T21:21:28.0940054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0940176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0940451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0940584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0940769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzt05o5bp
2023-01-11T21:21:28.0940963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzt05o5bp/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0941232Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0941241Z 
2023-01-11T21:21:28.0941313Z Running tests...
2023-01-11T21:21:28.0941506Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0941840Z   test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4116
2023-01-11T21:21:28.0941996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4117
2023-01-11T21:21:28.0942146Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4118
2023-01-11T21:21:28.0942296Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4119
2023-01-11T21:21:28.0942572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0942696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0942977Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0943114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0943285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnk8vx_5y
2023-01-11T21:21:28.0943480Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnk8vx_5y/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0943751Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0943873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0944152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0944285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0944467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkxksgv6u
2023-01-11T21:21:28.0944698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkxksgv6u/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0944966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0945073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0945351Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0945482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0945664Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf6mwa1ck
2023-01-11T21:21:28.0945858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf6mwa1ck/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0946124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0946278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0946556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0946688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0946859Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcs7twf6_
2023-01-11T21:21:28.0947050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcs7twf6_/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0947211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0947372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0947532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0947691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0947796Z skip: Need at least 2 CUDA devices (2.232s)
2023-01-11T21:21:28.0947802Z 
2023-01-11T21:21:28.0947996Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0948058Z Ran 1 test in 2.233s
2023-01-11T21:21:28.0948063Z 
2023-01-11T21:21:28.0948134Z OK (skipped=1)
2023-01-11T21:21:28.0948139Z 
2023-01-11T21:21:28.0948221Z Generating XML reports...
2023-01-11T21:21:28.0948638Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211810.xml
2023-01-11T21:21:28.0948908Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0949027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0949303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0949439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0949745Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyv7f6n8c
2023-01-11T21:21:28.0949929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyv7f6n8c/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0950199Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0950205Z 
2023-01-11T21:21:28.0950277Z Running tests...
2023-01-11T21:21:28.0950472Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0950754Z   test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4171
2023-01-11T21:21:28.0950911Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4172
2023-01-11T21:21:28.0951065Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4173
2023-01-11T21:21:28.0951247Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4174
2023-01-11T21:21:28.0951520Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0951628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0951909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0952042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0952225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qqw1qoh
2023-01-11T21:21:28.0952419Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qqw1qoh/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0952718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0952842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0953119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0953240Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0953424Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdsrj3d_q
2023-01-11T21:21:28.0953618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdsrj3d_q/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0953886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0954006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0954284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0954419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0954603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp62mmzpt0
2023-01-11T21:21:28.0954796Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp62mmzpt0/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0955052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0955170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0955449Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0955582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0955763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpme04b5mk
2023-01-11T21:21:28.0955956Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpme04b5mk/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0956119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0956279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0956427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0956587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0956689Z skip: Need at least 2 CUDA devices (2.233s)
2023-01-11T21:21:28.0956694Z 
2023-01-11T21:21:28.0956889Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0956965Z Ran 1 test in 2.233s
2023-01-11T21:21:28.0956970Z 
2023-01-11T21:21:28.0957040Z OK (skipped=1)
2023-01-11T21:21:28.0957044Z 
2023-01-11T21:21:28.0957131Z Generating XML reports...
2023-01-11T21:21:28.0957551Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211814.xml
2023-01-11T21:21:28.0957857Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0957968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0958247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0958380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0958562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdx9rha23
2023-01-11T21:21:28.0958754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdx9rha23/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0959020Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0959025Z 
2023-01-11T21:21:28.0959099Z Running tests...
2023-01-11T21:21:28.0959323Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0959601Z   test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4226
2023-01-11T21:21:28.0959744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4227
2023-01-11T21:21:28.0959895Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4228
2023-01-11T21:21:28.0960044Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4229
2023-01-11T21:21:28.0960316Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0960437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0960716Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0960855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0961041Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpli5s58ij
2023-01-11T21:21:28.0961224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpli5s58ij/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0961493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0961616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0961893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0962025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0962206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx3dol_b0
2023-01-11T21:21:28.0962400Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx3dol_b0/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0962669Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0962788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0963055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0963187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0963368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7a7v23xw
2023-01-11T21:21:28.0963638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0963758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0964039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0964203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0964386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5v1qx66d
2023-01-11T21:21:28.0964567Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7a7v23xw/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0964759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5v1qx66d/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0964922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0965082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0965243Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0965399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0965527Z skip: Need at least 1 CUDA device (2.329s)
2023-01-11T21:21:28.0965533Z 
2023-01-11T21:21:28.0965730Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0965806Z Ran 1 test in 2.329s
2023-01-11T21:21:28.0965812Z 
2023-01-11T21:21:28.0965870Z OK (skipped=1)
2023-01-11T21:21:28.0965875Z 
2023-01-11T21:21:28.0965958Z Generating XML reports...
2023-01-11T21:21:28.0966376Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211818.xml
2023-01-11T21:21:28.0966647Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0966769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0967046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0967181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0967367Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpphjj5vq3
2023-01-11T21:21:28.0967564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpphjj5vq3/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0967818Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0967823Z 
2023-01-11T21:21:28.0967895Z Running tests...
2023-01-11T21:21:28.0968089Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0968363Z   test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4281
2023-01-11T21:21:28.0968516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4282
2023-01-11T21:21:28.0968669Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4283
2023-01-11T21:21:28.0968823Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4284
2023-01-11T21:21:28.0969094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0969205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0969485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0969620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0969805Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprb_qlpbv
2023-01-11T21:21:28.0970000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprb_qlpbv/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0970269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0970393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0970717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0970957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0971127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgm2r79s8
2023-01-11T21:21:28.0971321Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgm2r79s8/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0971589Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0971712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0971993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0972127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0972357Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvk2ju_uz
2023-01-11T21:21:28.0972550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvk2ju_uz/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0972819Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0972929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0973207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0973342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0973526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5mxkpk26
2023-01-11T21:21:28.0973720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5mxkpk26/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0973889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0974053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0974214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0974361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0974465Z skip: Need at least 2 CUDA devices (2.330s)
2023-01-11T21:21:28.0974471Z 
2023-01-11T21:21:28.0974670Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0974745Z Ran 1 test in 2.330s
2023-01-11T21:21:28.0974750Z 
2023-01-11T21:21:28.0974821Z OK (skipped=1)
2023-01-11T21:21:28.0974826Z 
2023-01-11T21:21:28.0974911Z Generating XML reports...
2023-01-11T21:21:28.0975332Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211822.xml
2023-01-11T21:21:28.0975610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0975735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0976000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0976135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0976320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpywg1d6ez
2023-01-11T21:21:28.0976516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpywg1d6ez/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0976784Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0976790Z 
2023-01-11T21:21:28.0976862Z Running tests...
2023-01-11T21:21:28.0977061Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0977360Z   test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4336
2023-01-11T21:21:28.0977514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4337
2023-01-11T21:21:28.0977652Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4338
2023-01-11T21:21:28.0977805Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4339
2023-01-11T21:21:28.0978080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0978203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0978483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0978649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0978839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9315soaj
2023-01-11T21:21:28.0979033Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9315soaj/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0979289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0979412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0979688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0979824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0980007Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8obaanvm
2023-01-11T21:21:28.0980202Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8obaanvm/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0980476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0980598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0980878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0980999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0981183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi93kuh7c
2023-01-11T21:21:28.0981378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi93kuh7c/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0981717Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0981842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0982123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0982258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0982440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4kxgm7ye
2023-01-11T21:21:28.0982620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4kxgm7ye/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0982783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0982946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0983106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0983266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0983369Z skip: Need at least 2 CUDA devices (2.230s)
2023-01-11T21:21:28.0983422Z 
2023-01-11T21:21:28.0983620Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0983694Z Ran 1 test in 2.230s
2023-01-11T21:21:28.0983699Z 
2023-01-11T21:21:28.0983774Z OK (skipped=1)
2023-01-11T21:21:28.0983778Z 
2023-01-11T21:21:28.0983850Z Generating XML reports...
2023-01-11T21:21:28.0984267Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211826.xml
2023-01-11T21:21:28.0984542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0984663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0984942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0985074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0985288Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7h34120i
2023-01-11T21:21:28.0985485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7h34120i/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0985742Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0985760Z 
2023-01-11T21:21:28.0985820Z Running tests...
2023-01-11T21:21:28.0986015Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0986278Z   test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4391
2023-01-11T21:21:28.0986435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4392
2023-01-11T21:21:28.0986588Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4393
2023-01-11T21:21:28.0986737Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4394
2023-01-11T21:21:28.0987013Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0987135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0987403Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0987534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0987716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdkgi6147
2023-01-11T21:21:28.0987908Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdkgi6147/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0988177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0988298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0988582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0988717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0988901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi12k2jpr
2023-01-11T21:21:28.0989083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi12k2jpr/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0989355Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0989474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0989754Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0989885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0990067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_6dn8cor
2023-01-11T21:21:28.0990288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_6dn8cor/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0990556Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0990665Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0990944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0991078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0991259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ko1b_ty
2023-01-11T21:21:28.0991450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ko1b_ty/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0991641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.0991808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.0991970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.0992132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.0992224Z skip: Need at least 2 CUDA devices (2.329s)
2023-01-11T21:21:28.0992229Z 
2023-01-11T21:21:28.0992427Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0992505Z Ran 1 test in 2.329s
2023-01-11T21:21:28.0992511Z 
2023-01-11T21:21:28.0992583Z OK (skipped=1)
2023-01-11T21:21:28.0992587Z 
2023-01-11T21:21:28.0992672Z Generating XML reports...
2023-01-11T21:21:28.0993088Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211830.xml
2023-01-11T21:21:28.0993364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0993490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0993771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0993893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0994077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphhn14noc
2023-01-11T21:21:28.0994271Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphhn14noc/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0994538Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.0994543Z 
2023-01-11T21:21:28.0994616Z Running tests...
2023-01-11T21:21:28.0994812Z ----------------------------------------------------------------------
2023-01-11T21:21:28.0995097Z   test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4446
2023-01-11T21:21:28.0995254Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4447
2023-01-11T21:21:28.0995393Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4448
2023-01-11T21:21:28.0995543Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4449
2023-01-11T21:21:28.0995814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0995935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0996211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0996348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0996565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdv31ijrx
2023-01-11T21:21:28.0996759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdv31ijrx/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0997029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0997138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0997415Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0997549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0997730Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6krf7266
2023-01-11T21:21:28.0997923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6krf7266/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0998216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0998340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0998617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0998739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.0998921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27xc1mqe
2023-01-11T21:21:28.0999112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27xc1mqe/_remote_module_non_scriptable.py
2023-01-11T21:21:28.0999381Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.0999502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.0999780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.0999915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1000099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp02_wbuuf
2023-01-11T21:21:28.1000288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp02_wbuuf/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1000441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1000601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1000760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1000916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1001015Z skip: Need at least 1 CUDA device (2.326s)
2023-01-11T21:21:28.1001020Z 
2023-01-11T21:21:28.1001215Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1001295Z Ran 1 test in 2.326s
2023-01-11T21:21:28.1001300Z 
2023-01-11T21:21:28.1001368Z OK (skipped=1)
2023-01-11T21:21:28.1001372Z 
2023-01-11T21:21:28.1001458Z Generating XML reports...
2023-01-11T21:21:28.1001864Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211834.xml
2023-01-11T21:21:28.1002137Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1002257Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1002537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1002673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1002857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcivtwu84
2023-01-11T21:21:28.1003080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcivtwu84/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1003347Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1003352Z 
2023-01-11T21:21:28.1003426Z Running tests...
2023-01-11T21:21:28.1003609Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1003889Z   test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4501
2023-01-11T21:21:28.1004044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4502
2023-01-11T21:21:28.1004194Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4503
2023-01-11T21:21:28.1004344Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4504
2023-01-11T21:21:28.1004646Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1004768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1005050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1005171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1005354Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd1wjv9f1
2023-01-11T21:21:28.1005547Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd1wjv9f1/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1005815Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1005936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1006211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1006348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1006531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmponf__5wp
2023-01-11T21:21:28.1006726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmponf__5wp/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1006981Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1007106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1007382Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1007514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1007695Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpks_2auvs
2023-01-11T21:21:28.1007892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpks_2auvs/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1008158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1008280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1008544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1008679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1008863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd3qtd2hd
2023-01-11T21:21:28.1009055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd3qtd2hd/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1009215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1009409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1009568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1009728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1009826Z skip: Need at least 1 CUDA device (2.225s)
2023-01-11T21:21:28.1009832Z 
2023-01-11T21:21:28.1010016Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1010089Z Ran 1 test in 2.225s
2023-01-11T21:21:28.1010094Z 
2023-01-11T21:21:28.1010166Z OK (skipped=1)
2023-01-11T21:21:28.1010170Z 
2023-01-11T21:21:28.1010253Z Generating XML reports...
2023-01-11T21:21:28.1010669Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211838.xml
2023-01-11T21:21:28.1011107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1011236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1011517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1011637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1011819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeu_swdyj
2023-01-11T21:21:28.1012013Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeu_swdyj/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1012280Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1012285Z 
2023-01-11T21:21:28.1012357Z Running tests...
2023-01-11T21:21:28.1012555Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1012832Z   test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4556
2023-01-11T21:21:28.1012991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4557
2023-01-11T21:21:28.1013145Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4558
2023-01-11T21:21:28.1013282Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4559
2023-01-11T21:21:28.1013553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1013674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1013954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1014088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1014274Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4f2qnbw1
2023-01-11T21:21:28.1014470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4f2qnbw1/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1014742Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1014862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1015123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1015255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1015439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf8xbaxze
2023-01-11T21:21:28.1015633Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf8xbaxze/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1015902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1016060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1016339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1016474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1016644Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3i561k_a
2023-01-11T21:21:28.1016834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3i561k_a/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1017098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1017219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1017495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1017659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1017847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9xakg58v
2023-01-11T21:21:28.1018041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xakg58v/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1018204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1018352Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1018512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1018670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1018769Z skip: Need at least 1 CUDA device (2.227s)
2023-01-11T21:21:28.1018775Z 
2023-01-11T21:21:28.1018971Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1019052Z Ran 1 test in 2.227s
2023-01-11T21:21:28.1019057Z 
2023-01-11T21:21:28.1019126Z OK (skipped=1)
2023-01-11T21:21:28.1019131Z 
2023-01-11T21:21:28.1019213Z Generating XML reports...
2023-01-11T21:21:28.1019616Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211843.xml
2023-01-11T21:21:28.1019887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1020009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1020286Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1020418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1020602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu1ojiz1b
2023-01-11T21:21:28.1020800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu1ojiz1b/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1021068Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1021074Z 
2023-01-11T21:21:28.1021145Z Running tests...
2023-01-11T21:21:28.1021329Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1021656Z   test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4611
2023-01-11T21:21:28.1021815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4612
2023-01-11T21:21:28.1021967Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4613
2023-01-11T21:21:28.1022118Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4614
2023-01-11T21:21:28.1022395Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1022554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1022834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1022969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1023140Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz01yxg8d
2023-01-11T21:21:28.1023336Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz01yxg8d/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1023605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1023727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1024004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1024171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1024355Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7kwhrb1q
2023-01-11T21:21:28.1024550Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7kwhrb1q/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1024805Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1024926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1025202Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1025335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1025518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpap4fp9eg
2023-01-11T21:21:28.1025713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpap4fp9eg/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1025982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1026104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1026378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1026498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1026682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp516g8k4v
2023-01-11T21:21:28.1026870Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp516g8k4v/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1027033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1027195Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1027363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1027524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1027630Z skip: Need at least 2 CUDA devices (2.229s)
2023-01-11T21:21:28.1027635Z 
2023-01-11T21:21:28.1027834Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1027896Z Ran 1 test in 2.229s
2023-01-11T21:21:28.1027901Z 
2023-01-11T21:21:28.1027973Z OK (skipped=1)
2023-01-11T21:21:28.1027978Z 
2023-01-11T21:21:28.1028064Z Generating XML reports...
2023-01-11T21:21:28.1028483Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211847.xml
2023-01-11T21:21:28.1028758Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1028931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1029210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1029345Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1029515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2rx3osun
2023-01-11T21:21:28.1029710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2rx3osun/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1029978Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1029983Z 
2023-01-11T21:21:28.1030055Z Running tests...
2023-01-11T21:21:28.1030253Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1030553Z   test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4666
2023-01-11T21:21:28.1030711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4667
2023-01-11T21:21:28.1030864Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4668
2023-01-11T21:21:28.1031017Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4669
2023-01-11T21:21:28.1031279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1031402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1031682Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1031816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1032002Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvgog39yn
2023-01-11T21:21:28.1032200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvgog39yn/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1032475Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1032598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1032877Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1032997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1033181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp29ogphlc
2023-01-11T21:21:28.1033377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp29ogphlc/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1033645Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1033769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1034049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1034183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1034368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpboas9_5q
2023-01-11T21:21:28.1034549Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpboas9_5q/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1034817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1034939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1035217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1035351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1035568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpro7twtnd
2023-01-11T21:21:28.1035761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpro7twtnd/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1035925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1036087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1036236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1036393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1036494Z skip: Need at least 1 CUDA device (2.227s)
2023-01-11T21:21:28.1036500Z 
2023-01-11T21:21:28.1036698Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1036775Z Ran 1 test in 2.228s
2023-01-11T21:21:28.1036780Z 
2023-01-11T21:21:28.1036918Z OK (skipped=1)
2023-01-11T21:21:28.1036924Z 
2023-01-11T21:21:28.1037007Z Generating XML reports...
2023-01-11T21:21:28.1037428Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211851.xml
2023-01-11T21:21:28.1037688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1037814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1038092Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1038228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1038411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1yqnieay
2023-01-11T21:21:28.1038605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1yqnieay/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1038873Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1038878Z 
2023-01-11T21:21:28.1038952Z Running tests...
2023-01-11T21:21:28.1039148Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1039406Z   test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4721
2023-01-11T21:21:28.1039562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4722
2023-01-11T21:21:28.1039712Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4723
2023-01-11T21:21:28.1039863Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4724
2023-01-11T21:21:28.1040134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1040260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1040537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1040673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1040857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprwl2wlmq
2023-01-11T21:21:28.1041041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprwl2wlmq/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1041314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1041435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1041712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1041847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1042067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphqpvche4
2023-01-11T21:21:28.1042261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphqpvche4/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1042534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1042642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1042919Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1043051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1043234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0fubjize
2023-01-11T21:21:28.1043428Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0fubjize/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1043723Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1043848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1044124Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1044258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1044428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps95lql9l
2023-01-11T21:21:28.1044620Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps95lql9l/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1044783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1044944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1045107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1045271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1045369Z skip: Need at least 1 CUDA device (2.224s)
2023-01-11T21:21:28.1045374Z 
2023-01-11T21:21:28.1045573Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1045634Z Ran 1 test in 2.224s
2023-01-11T21:21:28.1045651Z 
2023-01-11T21:21:28.1045709Z OK (skipped=1)
2023-01-11T21:21:28.1045713Z 
2023-01-11T21:21:28.1045798Z Generating XML reports...
2023-01-11T21:21:28.1046217Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211855.xml
2023-01-11T21:21:28.1046491Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1046613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1046895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1047028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1047211Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqw1fp9zs
2023-01-11T21:21:28.1047391Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqw1fp9zs/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1047656Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1047661Z 
2023-01-11T21:21:28.1047735Z Running tests...
2023-01-11T21:21:28.1047935Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1048213Z   test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4776
2023-01-11T21:21:28.1048400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4777
2023-01-11T21:21:28.1048552Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4778
2023-01-11T21:21:28.1048701Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4779
2023-01-11T21:21:28.1048972Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1049083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1049361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1049496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1049679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwehh6rq8
2023-01-11T21:21:28.1049898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwehh6rq8/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1050169Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1050292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1050566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1050687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1050984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprndxxp0m
2023-01-11T21:21:28.1051181Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprndxxp0m/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1051452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1051576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1051859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1051994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1052178Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7oaphxa1
2023-01-11T21:21:28.1052372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7oaphxa1/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1052632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1052755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1053037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1053173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1053361Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8y4u9b9
2023-01-11T21:21:28.1053559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8y4u9b9/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1053724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1053889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1054039Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1054205Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1054306Z skip: Need at least 1 CUDA device (2.326s)
2023-01-11T21:21:28.1054311Z 
2023-01-11T21:21:28.1054509Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1054586Z Ran 1 test in 2.326s
2023-01-11T21:21:28.1054591Z 
2023-01-11T21:21:28.1054664Z OK (skipped=1)
2023-01-11T21:21:28.1054717Z 
2023-01-11T21:21:28.1054807Z Generating XML reports...
2023-01-11T21:21:28.1055228Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211859.xml
2023-01-11T21:21:28.1055501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1055612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1055892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1056027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1056212Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq4zvqqwc
2023-01-11T21:21:28.1056411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq4zvqqwc/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1056721Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1056729Z 
2023-01-11T21:21:28.1056802Z Running tests...
2023-01-11T21:21:28.1056999Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1057279Z   test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4831
2023-01-11T21:21:28.1057423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4832
2023-01-11T21:21:28.1057577Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4833
2023-01-11T21:21:28.1057724Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4834
2023-01-11T21:21:28.1057997Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1058116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1058398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1058532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1058712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphwzhs_8m
2023-01-11T21:21:28.1058894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphwzhs_8m/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1059166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1059286Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1059563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1059698Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1059885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4wvrr6wk
2023-01-11T21:21:28.1060082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4wvrr6wk/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1060352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1060473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1060738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1060871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1061051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_3dvtfj
2023-01-11T21:21:28.1061245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_3dvtfj/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1061514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1061738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1062022Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1062156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1062338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_o52uedr
2023-01-11T21:21:28.1062514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_o52uedr/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1062677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1062840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1062999Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1063192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1063296Z skip: Need at least 1 CUDA device (2.230s)
2023-01-11T21:21:28.1063301Z 
2023-01-11T21:21:28.1063499Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1063576Z Ran 1 test in 2.231s
2023-01-11T21:21:28.1063582Z 
2023-01-11T21:21:28.1063640Z OK (skipped=1)
2023-01-11T21:21:28.1063659Z 
2023-01-11T21:21:28.1063730Z Generating XML reports...
2023-01-11T21:21:28.1064149Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211903.xml
2023-01-11T21:21:28.1064420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1064544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1064825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1064960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1065144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb10jm6yy
2023-01-11T21:21:28.1065339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb10jm6yy/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1065593Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1065599Z 
2023-01-11T21:21:28.1065672Z Running tests...
2023-01-11T21:21:28.1065867Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1066155Z   test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4886
2023-01-11T21:21:28.1066311Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4887
2023-01-11T21:21:28.1066462Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4888
2023-01-11T21:21:28.1066610Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4889
2023-01-11T21:21:28.1066882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1067004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1067272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1067406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1067587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr3_g10ca
2023-01-11T21:21:28.1067777Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr3_g10ca/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1068078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1068205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1068479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1068614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1068786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7bwpmcwl
2023-01-11T21:21:28.1068980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7bwpmcwl/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1069245Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1069367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1069676Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1069811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1069992Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi7zj3rrn
2023-01-11T21:21:28.1070183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi7zj3rrn/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1070451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1070563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1070837Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1070968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1071152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphe6zq31w
2023-01-11T21:21:28.1071345Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphe6zq31w/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1071506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1071665Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1071824Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1071973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1072074Z skip: Need at least 1 CUDA device (2.328s)
2023-01-11T21:21:28.1072080Z 
2023-01-11T21:21:28.1072276Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1072354Z Ran 1 test in 2.329s
2023-01-11T21:21:28.1072359Z 
2023-01-11T21:21:28.1072429Z OK (skipped=1)
2023-01-11T21:21:28.1072433Z 
2023-01-11T21:21:28.1072518Z Generating XML reports...
2023-01-11T21:21:28.1072938Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211907.xml
2023-01-11T21:21:28.1073214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1073336Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1073604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1073739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1073921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpni030igj
2023-01-11T21:21:28.1074113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpni030igj/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1074382Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1074430Z 
2023-01-11T21:21:28.1074501Z Running tests...
2023-01-11T21:21:28.1074696Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1074974Z   test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4941
2023-01-11T21:21:28.1075126Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4942
2023-01-11T21:21:28.1075265Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4943
2023-01-11T21:21:28.1075417Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4944
2023-01-11T21:21:28.1075688Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1075809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1076118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1076255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1076439Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_03sda1
2023-01-11T21:21:28.1076628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_03sda1/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1076887Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1077007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1077289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1077426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1077611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuk61imly
2023-01-11T21:21:28.1077880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1078002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1078280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1078413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1078585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppwf42tbp
2023-01-11T21:21:28.1078780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuk61imly/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1078970Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppwf42tbp/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1079239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1079360Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1079633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1079765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1079945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzemzbfcb
2023-01-11T21:21:28.1080137Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzemzbfcb/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1080287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1080448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1080609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1080807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1080906Z skip: Need at least 1 CUDA device (2.235s)
2023-01-11T21:21:28.1080912Z 
2023-01-11T21:21:28.1081107Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1081182Z Ran 1 test in 2.235s
2023-01-11T21:21:28.1081187Z 
2023-01-11T21:21:28.1081258Z OK (skipped=1)
2023-01-11T21:21:28.1081263Z 
2023-01-11T21:21:28.1081333Z Generating XML reports...
2023-01-11T21:21:28.1081753Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211911.xml
2023-01-11T21:21:28.1082028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1082155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1082465Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1082604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1082791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqlmqhvyp
2023-01-11T21:21:28.1082987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqlmqhvyp/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1083259Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1083264Z 
2023-01-11T21:21:28.1083324Z Running tests...
2023-01-11T21:21:28.1083522Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1083808Z   test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4996
2023-01-11T21:21:28.1083963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4997
2023-01-11T21:21:28.1084122Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 4998
2023-01-11T21:21:28.1084273Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 4999
2023-01-11T21:21:28.1084547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1084670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1084949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1085070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1085256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl1muqjw3
2023-01-11T21:21:28.1085452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl1muqjw3/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1085724Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1085849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1086125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1086265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1086448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6de_d_98
2023-01-11T21:21:28.1086627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6de_d_98/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1086896Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1087017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1087300Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1087471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1087654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5jej29lv
2023-01-11T21:21:28.1087848Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5jej29lv/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1088115Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1088240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1088504Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1088637Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1088823Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprg0j_08s
2023-01-11T21:21:28.1089044Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprg0j_08s/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1089212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1089375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1089537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1089698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1089785Z skip: Need at least 1 CUDA device (2.229s)
2023-01-11T21:21:28.1089804Z 
2023-01-11T21:21:28.1089990Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1090066Z Ran 1 test in 2.229s
2023-01-11T21:21:28.1090070Z 
2023-01-11T21:21:28.1090143Z OK (skipped=1)
2023-01-11T21:21:28.1090147Z 
2023-01-11T21:21:28.1090233Z Generating XML reports...
2023-01-11T21:21:28.1090655Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211915.xml
2023-01-11T21:21:28.1091042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1091167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1091446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1091567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1091754Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprimtxpyt
2023-01-11T21:21:28.1091948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprimtxpyt/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1092215Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1092225Z 
2023-01-11T21:21:28.1092298Z Running tests...
2023-01-11T21:21:28.1092497Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1092763Z   test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5051
2023-01-11T21:21:28.1092918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5052
2023-01-11T21:21:28.1093071Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5053
2023-01-11T21:21:28.1093210Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5054
2023-01-11T21:21:28.1093483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1093605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1093885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1094070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1094254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplqb7ik09
2023-01-11T21:21:28.1094450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplqb7ik09/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1094722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1094830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1095110Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1095244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1095428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpevhx348v
2023-01-11T21:21:28.1095663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpevhx348v/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1095937Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1096059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1096342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1096474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1096642Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpswlefjta
2023-01-11T21:21:28.1096837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpswlefjta/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1097105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1097232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1097510Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1097641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1097824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbuio7lyx
2023-01-11T21:21:28.1098016Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbuio7lyx/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1098166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1098327Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1098488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1098656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1098763Z skip: Need at least 2 CUDA devices (2.237s)
2023-01-11T21:21:28.1098768Z 
2023-01-11T21:21:28.1098966Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1099042Z Ran 1 test in 2.238s
2023-01-11T21:21:28.1099047Z 
2023-01-11T21:21:28.1099119Z OK (skipped=1)
2023-01-11T21:21:28.1099124Z 
2023-01-11T21:21:28.1099209Z Generating XML reports...
2023-01-11T21:21:28.1099614Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211919.xml
2023-01-11T21:21:28.1099883Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1100004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1100284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1100454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1100638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjt4av0g6
2023-01-11T21:21:28.1100832Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjt4av0g6/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1101102Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1101107Z 
2023-01-11T21:21:28.1101181Z Running tests...
2023-01-11T21:21:28.1101363Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1101706Z   test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5106
2023-01-11T21:21:28.1101865Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5107
2023-01-11T21:21:28.1102051Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5108
2023-01-11T21:21:28.1102206Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5109
2023-01-11T21:21:28.1102484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1102609Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1102895Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1103016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1103201Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpas2xb9l0
2023-01-11T21:21:28.1103394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpas2xb9l0/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1103666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1103792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1104068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1104202Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1104385Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnd6v27nb
2023-01-11T21:21:28.1104578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnd6v27nb/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1104834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1104955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1105233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1105368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1105551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphb_3clqd
2023-01-11T21:21:28.1105744Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphb_3clqd/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1106012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1106135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1106401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1106537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1106718Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5blbxv4
2023-01-11T21:21:28.1106911Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5blbxv4/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1107107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1107268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1107427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1107587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1107689Z skip: Need at least 2 CUDA devices (2.229s)
2023-01-11T21:21:28.1107694Z 
2023-01-11T21:21:28.1107877Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1107953Z Ran 1 test in 2.230s
2023-01-11T21:21:28.1107958Z 
2023-01-11T21:21:28.1108028Z OK (skipped=1)
2023-01-11T21:21:28.1108032Z 
2023-01-11T21:21:28.1108115Z Generating XML reports...
2023-01-11T21:21:28.1108561Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211923.xml
2023-01-11T21:21:28.1108833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1108958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1109237Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1109373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1109544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm21alwg2
2023-01-11T21:21:28.1109738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm21alwg2/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1110005Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1110011Z 
2023-01-11T21:21:28.1110085Z Running tests...
2023-01-11T21:21:28.1110283Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1110551Z   test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5161
2023-01-11T21:21:28.1110707Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5162
2023-01-11T21:21:28.1110856Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5163
2023-01-11T21:21:28.1110994Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5164
2023-01-11T21:21:28.1111265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1111389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1111670Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1111806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1111989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ga9to5l
2023-01-11T21:21:28.1112183Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ga9to5l/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1112452Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1112574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1112838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1112970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1113154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgtji_uqa
2023-01-11T21:21:28.1113349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgtji_uqa/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1113649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1113774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1114054Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1114187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1114373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8zvpq8jd
2023-01-11T21:21:28.1114555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8zvpq8jd/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1114822Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1114946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1115251Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1115386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1115568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ncbm0fa
2023-01-11T21:21:28.1115761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ncbm0fa/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1115924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1116075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1116236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1116397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1116504Z skip: Need at least 2 CUDA devices (2.326s)
2023-01-11T21:21:28.1116511Z 
2023-01-11T21:21:28.1116710Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1116785Z Ran 1 test in 2.326s
2023-01-11T21:21:28.1116791Z 
2023-01-11T21:21:28.1116863Z OK (skipped=1)
2023-01-11T21:21:28.1116867Z 
2023-01-11T21:21:28.1116953Z Generating XML reports...
2023-01-11T21:21:28.1117366Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211927.xml
2023-01-11T21:21:28.1117625Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1117746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1118027Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1118159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1118347Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt1k00h2h
2023-01-11T21:21:28.1118540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt1k00h2h/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1118804Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1118809Z 
2023-01-11T21:21:28.1118882Z Running tests...
2023-01-11T21:21:28.1119063Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1119324Z   test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5216
2023-01-11T21:21:28.1119477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5217
2023-01-11T21:21:28.1119630Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5218
2023-01-11T21:21:28.1119781Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5219
2023-01-11T21:21:28.1120094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1120215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1120495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1120628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1120802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp505po7vj
2023-01-11T21:21:28.1121069Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1121192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1121496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1121634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1121816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo34lakha
2023-01-11T21:21:28.1122010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp505po7vj/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1122201Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo34lakha/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1122474Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1122582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1122860Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1122994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1123181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzt1r9mc6
2023-01-11T21:21:28.1123375Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzt1r9mc6/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1123648Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1123771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1124048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1124167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1124350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjr9qwpcj
2023-01-11T21:21:28.1124542Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjr9qwpcj/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1124705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1124869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1125026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1125186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1125288Z skip: Need at least 2 CUDA devices (2.226s)
2023-01-11T21:21:28.1125293Z 
2023-01-11T21:21:28.1125488Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1125552Z Ran 1 test in 2.226s
2023-01-11T21:21:28.1125557Z 
2023-01-11T21:21:28.1125626Z OK (skipped=1)
2023-01-11T21:21:28.1125631Z 
2023-01-11T21:21:28.1125713Z Generating XML reports...
2023-01-11T21:21:28.1126126Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211932.xml
2023-01-11T21:21:28.1126427Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1126552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1126831Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1126962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1127134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz5kjosjp
2023-01-11T21:21:28.1127329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz5kjosjp/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1127596Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1127601Z 
2023-01-11T21:21:28.1127673Z Running tests...
2023-01-11T21:21:28.1127866Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1128169Z   test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5271
2023-01-11T21:21:28.1128326Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5272
2023-01-11T21:21:28.1128477Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5273
2023-01-11T21:21:28.1128626Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5274
2023-01-11T21:21:28.1128886Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1129008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1129285Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1129417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1129603Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3n_2w638
2023-01-11T21:21:28.1129792Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3n_2w638/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1130063Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1130188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1130463Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1130584Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1130767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvkduprex
2023-01-11T21:21:28.1131076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvkduprex/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1131352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1131474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1131752Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1131887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1132073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx6v51cmi
2023-01-11T21:21:28.1132251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx6v51cmi/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1132518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1132642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1132924Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1133106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1133291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnxlltdwg
2023-01-11T21:21:28.1133485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnxlltdwg/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1133649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1133814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1133961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1134122Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1134224Z skip: Need at least 4 CUDA devices (2.224s)
2023-01-11T21:21:28.1134229Z 
2023-01-11T21:21:28.1134464Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1134542Z Ran 1 test in 2.224s
2023-01-11T21:21:28.1134547Z 
2023-01-11T21:21:28.1134619Z OK (skipped=1)
2023-01-11T21:21:28.1134623Z 
2023-01-11T21:21:28.1134707Z Generating XML reports...
2023-01-11T21:21:28.1135127Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211936.xml
2023-01-11T21:21:28.1135401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1135511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1135793Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1135926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1136110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpki6xbxc7
2023-01-11T21:21:28.1136306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpki6xbxc7/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1136573Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1136578Z 
2023-01-11T21:21:28.1136651Z Running tests...
2023-01-11T21:21:28.1136848Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1137109Z   test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5326
2023-01-11T21:21:28.1137265Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5327
2023-01-11T21:21:28.1137417Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5328
2023-01-11T21:21:28.1137569Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5329
2023-01-11T21:21:28.1137845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1137967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1138248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1138384Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1138569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5c3pn06r
2023-01-11T21:21:28.1138753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5c3pn06r/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1139024Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1139146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1139425Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1139628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1139812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_mdx35s
2023-01-11T21:21:28.1140006Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_mdx35s/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1140278Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1140387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1140664Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1140799Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1141013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw7w1hk6p
2023-01-11T21:21:28.1141215Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw7w1hk6p/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1141484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1141677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1141962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1142093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1142263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgagpqsbz
2023-01-11T21:21:28.1142460Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgagpqsbz/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1142623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1142787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1142948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1143109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1143213Z skip: Need at least 4 CUDA devices (2.425s)
2023-01-11T21:21:28.1143218Z 
2023-01-11T21:21:28.1143416Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1143479Z Ran 1 test in 2.425s
2023-01-11T21:21:28.1143498Z 
2023-01-11T21:21:28.1143556Z OK (skipped=1)
2023-01-11T21:21:28.1143561Z 
2023-01-11T21:21:28.1143647Z Generating XML reports...
2023-01-11T21:21:28.1144067Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211940.xml
2023-01-11T21:21:28.1144345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1144476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1144757Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1144893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1145080Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmfhd4vt9
2023-01-11T21:21:28.1145262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmfhd4vt9/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1145528Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1145533Z 
2023-01-11T21:21:28.1145607Z Running tests...
2023-01-11T21:21:28.1145806Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1146085Z   test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5381
2023-01-11T21:21:28.1146279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5382
2023-01-11T21:21:28.1146432Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5383
2023-01-11T21:21:28.1146586Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5384
2023-01-11T21:21:28.1146862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1146973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1147256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1147393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1147611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3mac3w6p
2023-01-11T21:21:28.1147814Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3mac3w6p/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1148087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1148209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1148484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1148617Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1148787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4rp3bli
2023-01-11T21:21:28.1148979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4rp3bli/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1149254Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1149379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1149656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1149792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1149974Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7pc1n6sr
2023-01-11T21:21:28.1150167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7pc1n6sr/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1150423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1150543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1150818Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1150955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1151139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpco7ed8bd
2023-01-11T21:21:28.1151329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpco7ed8bd/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1151489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1151651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1151810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1151959Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1152062Z skip: Need at least 2 CUDA devices (2.224s)
2023-01-11T21:21:28.1152067Z 
2023-01-11T21:21:28.1152261Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1152366Z Ran 1 test in 2.224s
2023-01-11T21:21:28.1152371Z 
2023-01-11T21:21:28.1152441Z OK (skipped=1)
2023-01-11T21:21:28.1152445Z 
2023-01-11T21:21:28.1152527Z Generating XML reports...
2023-01-11T21:21:28.1152943Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211944.xml
2023-01-11T21:21:28.1153216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1153324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1153599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1153734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1153918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsp6mrvk9
2023-01-11T21:21:28.1154140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsp6mrvk9/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1154408Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1154413Z 
2023-01-11T21:21:28.1154485Z Running tests...
2023-01-11T21:21:28.1154680Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1154937Z   test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5436
2023-01-11T21:21:28.1155079Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5437
2023-01-11T21:21:28.1155229Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5438
2023-01-11T21:21:28.1155378Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5439
2023-01-11T21:21:28.1155649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1155772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1156049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1156181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1156366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwnrz8jnb
2023-01-11T21:21:28.1156562Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwnrz8jnb/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1156823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1156946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1157221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1157355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1157539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8da74wld
2023-01-11T21:21:28.1157731Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8da74wld/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1157998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1158117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1158384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1158514Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1158697Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnnoxrvys
2023-01-11T21:21:28.1158893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnnoxrvys/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1159192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1159313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1159588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1159718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1159901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb0x2s2eo
2023-01-11T21:21:28.1160080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb0x2s2eo/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1160242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1160429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1160593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1160750Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1160849Z skip: Need at least 1 CUDA device (2.324s)
2023-01-11T21:21:28.1160854Z 
2023-01-11T21:21:28.1161051Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1161124Z Ran 1 test in 2.324s
2023-01-11T21:21:28.1161129Z 
2023-01-11T21:21:28.1161187Z OK (skipped=1)
2023-01-11T21:21:28.1161204Z 
2023-01-11T21:21:28.1161275Z Generating XML reports...
2023-01-11T21:21:28.1161690Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211948.xml
2023-01-11T21:21:28.1161959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1162085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1162364Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1162497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1162680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl1nu9occ
2023-01-11T21:21:28.1162871Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl1nu9occ/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1163123Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1163140Z 
2023-01-11T21:21:28.1163199Z Running tests...
2023-01-11T21:21:28.1163393Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1163660Z   test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5491
2023-01-11T21:21:28.1163814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5492
2023-01-11T21:21:28.1163964Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5493
2023-01-11T21:21:28.1164114Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5494
2023-01-11T21:21:28.1164387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1164507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1164774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1164908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1165092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiz8mfbhs
2023-01-11T21:21:28.1165334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiz8mfbhs/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1165608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1165729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1166006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1166142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1166312Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl0xh3lvo
2023-01-11T21:21:28.1166579Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1166701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1167010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1167146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1167328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa2g6_190
2023-01-11T21:21:28.1167523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl0xh3lvo/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1167710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa2g6_190/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1167980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1168087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1168365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1168500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1168684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcrqz2z0e
2023-01-11T21:21:28.1168876Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcrqz2z0e/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1169038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1169198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1169357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1169506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1169607Z skip: Need at least 2 CUDA devices (2.228s)
2023-01-11T21:21:28.1169613Z 
2023-01-11T21:21:28.1169808Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1169881Z Ran 1 test in 2.228s
2023-01-11T21:21:28.1169889Z 
2023-01-11T21:21:28.1169961Z OK (skipped=1)
2023-01-11T21:21:28.1169965Z 
2023-01-11T21:21:28.1170052Z Generating XML reports...
2023-01-11T21:21:28.1170469Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211952.xml
2023-01-11T21:21:28.1170741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1170972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1171240Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1171373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1171557Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9y3cujkc
2023-01-11T21:21:28.1171753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9y3cujkc/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1172074Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1172079Z 
2023-01-11T21:21:28.1172152Z Running tests...
2023-01-11T21:21:28.1172347Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1172619Z   test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5546
2023-01-11T21:21:28.1172773Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5547
2023-01-11T21:21:28.1172913Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5548
2023-01-11T21:21:28.1173063Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5549
2023-01-11T21:21:28.1173332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1173494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1173774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1173909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1174093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8pgxuv1
2023-01-11T21:21:28.1174291Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8pgxuv1/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1174550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1174675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1174950Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1175088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1175273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpurhmevb3
2023-01-11T21:21:28.1175538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1175658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1175938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1176069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1176242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmlvlswm_
2023-01-11T21:21:28.1176438Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpurhmevb3/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1176625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmlvlswm_/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1176897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1177020Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1177297Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1177431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1177613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9zbceff4
2023-01-11T21:21:28.1177802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9zbceff4/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1177953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1178115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1178311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1178469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1178570Z skip: Need at least 2 CUDA devices (2.230s)
2023-01-11T21:21:28.1178575Z 
2023-01-11T21:21:28.1178771Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1178845Z Ran 1 test in 2.230s
2023-01-11T21:21:28.1178849Z 
2023-01-11T21:21:28.1178919Z OK (skipped=1)
2023-01-11T21:21:28.1178924Z 
2023-01-11T21:21:28.1178994Z Generating XML reports...
2023-01-11T21:21:28.1179411Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211956.xml
2023-01-11T21:21:28.1179683Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1179804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1180112Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1180249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1180431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdgdcu86u
2023-01-11T21:21:28.1180624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdgdcu86u/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1180890Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1180896Z 
2023-01-11T21:21:28.1180955Z Running tests...
2023-01-11T21:21:28.1181149Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1181426Z   test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5601
2023-01-11T21:21:28.1181650Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5602
2023-01-11T21:21:28.1181807Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5603
2023-01-11T21:21:28.1181957Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5604
2023-01-11T21:21:28.1182233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1182356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1182638Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1182758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1182943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpft2f8_
2023-01-11T21:21:28.1183140Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpft2f8_/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1183410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1183533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1183811Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1183946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1184129Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxea7ndyi
2023-01-11T21:21:28.1184312Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxea7ndyi/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1184582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1184703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1185021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1185153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1185336Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqxp10ud
2023-01-11T21:21:28.1185528Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqxp10ud/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1185795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1185915Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1186178Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1186312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1186522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbl0b121v
2023-01-11T21:21:28.1186713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbl0b121v/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1186874Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1187036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1187198Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1187358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1187444Z skip: Need at least 1 CUDA device (2.232s)
2023-01-11T21:21:28.1187460Z 
2023-01-11T21:21:28.1187643Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1187718Z Ran 1 test in 2.232s
2023-01-11T21:21:28.1187723Z 
2023-01-11T21:21:28.1187792Z OK (skipped=1)
2023-01-11T21:21:28.1187801Z 
2023-01-11T21:21:28.1187884Z Generating XML reports...
2023-01-11T21:21:28.1188298Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212000.xml
2023-01-11T21:21:28.1188569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1188689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1188966Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1189087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1189269Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp799ao8sj
2023-01-11T21:21:28.1189459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp799ao8sj/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1189731Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1189736Z 
2023-01-11T21:21:28.1189807Z Running tests...
2023-01-11T21:21:28.1190001Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1190279Z   test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5656
2023-01-11T21:21:28.1190435Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5657
2023-01-11T21:21:28.1190586Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5658
2023-01-11T21:21:28.1190723Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5659
2023-01-11T21:21:28.1190996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1191117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1191428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1191564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1191750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfz6bwx6g
2023-01-11T21:21:28.1191944Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfz6bwx6g/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1192217Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1192326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1192602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1192736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1192951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpejtz1dh7
2023-01-11T21:21:28.1193148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpejtz1dh7/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1193419Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1193540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1193820Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1193955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1194125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3laum6h4
2023-01-11T21:21:28.1194320Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3laum6h4/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1194591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1194717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1194996Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1195129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1195310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5zekorhh
2023-01-11T21:21:28.1195503Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5zekorhh/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1195654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1195816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1195980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1196142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1196245Z skip: Need at least 2 CUDA devices (2.329s)
2023-01-11T21:21:28.1196250Z 
2023-01-11T21:21:28.1196445Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1196522Z Ran 1 test in 2.329s
2023-01-11T21:21:28.1196527Z 
2023-01-11T21:21:28.1196599Z OK (skipped=1)
2023-01-11T21:21:28.1196604Z 
2023-01-11T21:21:28.1196689Z Generating XML reports...
2023-01-11T21:21:28.1197090Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212004.xml
2023-01-11T21:21:28.1197363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1197487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1197767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1197932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1198118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdoo3yv3i
2023-01-11T21:21:28.1198313Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdoo3yv3i/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1198582Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1198587Z 
2023-01-11T21:21:28.1198661Z Running tests...
2023-01-11T21:21:28.1198843Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1199123Z   test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5711
2023-01-11T21:21:28.1199308Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5712
2023-01-11T21:21:28.1199464Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5713
2023-01-11T21:21:28.1199615Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5714
2023-01-11T21:21:28.1199889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1200016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1200296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1200417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1200601Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprlluz3cz
2023-01-11T21:21:28.1200794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprlluz3cz/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1201068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1201191Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1201468Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1201601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1201782Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3_k1cgkf
2023-01-11T21:21:28.1201973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3_k1cgkf/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1202228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1202348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1202627Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1202762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1202942Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvw046md7
2023-01-11T21:21:28.1203134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvw046md7/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1203404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1203526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1203802Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1203922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1204108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp94j1zdwe
2023-01-11T21:21:28.1204329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp94j1zdwe/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1204491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1204652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1204809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1204969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1205073Z skip: Need at least 2 CUDA devices (2.231s)
2023-01-11T21:21:28.1205078Z 
2023-01-11T21:21:28.1205264Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1205337Z Ran 1 test in 2.231s
2023-01-11T21:21:28.1205343Z 
2023-01-11T21:21:28.1205413Z OK (skipped=1)
2023-01-11T21:21:28.1205418Z 
2023-01-11T21:21:28.1205531Z Generating XML reports...
2023-01-11T21:21:28.1205952Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212008.xml
2023-01-11T21:21:28.1206223Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1206344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1206622Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1206755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1206927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpck86ikj4
2023-01-11T21:21:28.1207123Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpck86ikj4/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1207392Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1207398Z 
2023-01-11T21:21:28.1207471Z Running tests...
2023-01-11T21:21:28.1207665Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1207945Z   test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5766
2023-01-11T21:21:28.1208099Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5767
2023-01-11T21:21:28.1208249Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5768
2023-01-11T21:21:28.1208399Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5769
2023-01-11T21:21:28.1208660Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1208783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1209068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1209200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1209383Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1dsl2f7q
2023-01-11T21:21:28.1209577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1dsl2f7q/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1209846Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1209968Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1210233Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1210364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1210549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp14ma8d2m
2023-01-11T21:21:28.1210783Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp14ma8d2m/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1211166Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1211288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1211571Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1211706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1211889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiuynl_fu
2023-01-11T21:21:28.1212067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiuynl_fu/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1212386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1212511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1212789Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1212920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1213102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprla_9sis
2023-01-11T21:21:28.1213292Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprla_9sis/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1213455Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1213605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1213767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1213932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1214036Z skip: Need at least 2 CUDA devices (2.331s)
2023-01-11T21:21:28.1214042Z 
2023-01-11T21:21:28.1214240Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1214315Z Ran 1 test in 2.331s
2023-01-11T21:21:28.1214320Z 
2023-01-11T21:21:28.1214391Z OK (skipped=1)
2023-01-11T21:21:28.1214396Z 
2023-01-11T21:21:28.1214480Z Generating XML reports...
2023-01-11T21:21:28.1214896Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212013.xml
2023-01-11T21:21:28.1215152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1215272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1215550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1215684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1215863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe7o_k7ln
2023-01-11T21:21:28.1216056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe7o_k7ln/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1216320Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1216326Z 
2023-01-11T21:21:28.1216397Z Running tests...
2023-01-11T21:21:28.1216592Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1216852Z   test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5821
2023-01-11T21:21:28.1217006Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5822
2023-01-11T21:21:28.1217198Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5823
2023-01-11T21:21:28.1217347Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5824
2023-01-11T21:21:28.1217619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1217739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1218018Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1218152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1218323Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7abyhhvs
2023-01-11T21:21:28.1218520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7abyhhvs/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1218817Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1218941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1219214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1219348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1219531Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpig82jg3m
2023-01-11T21:21:28.1219723Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpig82jg3m/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1219989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1220097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1220377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1220511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1220693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8cdge85m
2023-01-11T21:21:28.1220885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8cdge85m/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1221151Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1221272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1221552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1221739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1221922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw11u0ra8
2023-01-11T21:21:28.1222121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw11u0ra8/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1222284Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1222446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1222607Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1222770Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1222870Z skip: Need at least 1 CUDA device (2.230s)
2023-01-11T21:21:28.1222875Z 
2023-01-11T21:21:28.1223076Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1223138Z Ran 1 test in 2.230s
2023-01-11T21:21:28.1223143Z 
2023-01-11T21:21:28.1223215Z OK (skipped=1)
2023-01-11T21:21:28.1223219Z 
2023-01-11T21:21:28.1223303Z Generating XML reports...
2023-01-11T21:21:28.1223768Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212017.xml
2023-01-11T21:21:28.1224039Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1224161Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1224440Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1224576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1224759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplf6q3ma_
2023-01-11T21:21:28.1224941Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplf6q3ma_/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1225205Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1225242Z 
2023-01-11T21:21:28.1225315Z Running tests...
2023-01-11T21:21:28.1225512Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1225783Z   test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5876
2023-01-11T21:21:28.1225936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5877
2023-01-11T21:21:28.1226086Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5878
2023-01-11T21:21:28.1226236Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5879
2023-01-11T21:21:28.1226495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1226615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1226892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1227028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1227212Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpake7ryfh
2023-01-11T21:21:28.1227408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpake7ryfh/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1227675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1227796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1228073Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1228194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1228377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpstl12wr0
2023-01-11T21:21:28.1228573Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpstl12wr0/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1228838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1229127Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1229481Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1229645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1229857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpolgwf_bo
2023-01-11T21:21:28.1230088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpolgwf_bo/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1230386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1230528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1230836Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1231002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1231241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6vml5g03
2023-01-11T21:21:28.1231459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6vml5g03/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1231659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1231851Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1232038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1232215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1232349Z skip: Need at least 2 CUDA devices (2.231s)
2023-01-11T21:21:28.1232355Z 
2023-01-11T21:21:28.1232580Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1232685Z Ran 1 test in 2.231s
2023-01-11T21:21:28.1232690Z 
2023-01-11T21:21:28.1232845Z OK (skipped=1)
2023-01-11T21:21:28.1232850Z 
2023-01-11T21:21:28.1232972Z Generating XML reports...
2023-01-11T21:21:28.1233416Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212021.xml
2023-01-11T21:21:28.1233718Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1233870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1234134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1234305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1234516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu_f8mted
2023-01-11T21:21:28.1234743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu_f8mted/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1235064Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1235070Z 
2023-01-11T21:21:28.1235172Z Running tests...
2023-01-11T21:21:28.1235396Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1236166Z   test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81962 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.304s)
2023-01-11T21:21:28.1236174Z 
2023-01-11T21:21:28.1236394Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1236499Z Ran 1 test in 0.304s
2023-01-11T21:21:28.1236504Z 
2023-01-11T21:21:28.1236562Z OK (skipped=1)
2023-01-11T21:21:28.1236567Z 
2023-01-11T21:21:28.1236680Z Generating XML reports...
2023-01-11T21:21:28.1237163Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212025.xml
2023-01-11T21:21:28.1237484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1237637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1237945Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1238113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1238364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmped20ewwe
2023-01-11T21:21:28.1238595Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmped20ewwe/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1238849Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1238854Z 
2023-01-11T21:21:28.1238956Z Running tests...
2023-01-11T21:21:28.1239180Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1239500Z   test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5941
2023-01-11T21:21:28.1239685Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5942
2023-01-11T21:21:28.1239866Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5943
2023-01-11T21:21:28.1240076Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5944
2023-01-11T21:21:28.1240394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1240505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1240813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1241008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1241221Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfss57add
2023-01-11T21:21:28.1241470Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfss57add/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1241772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1241926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1242248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1242409Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1242580Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf2s2xv9h
2023-01-11T21:21:28.1242802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf2s2xv9h/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1243101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1243251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1243577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1243740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1243961Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfv9yc3ba
2023-01-11T21:21:28.1244180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfv9yc3ba/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1244477Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1244586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1244922Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1245084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1245295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpktz4ppy5
2023-01-11T21:21:28.1245535Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpktz4ppy5/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1245766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1245954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1246144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1246292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1246425Z skip: Need at least 2 CUDA devices (2.328s)
2023-01-11T21:21:28.1246430Z 
2023-01-11T21:21:28.1246656Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1256946Z Ran 1 test in 2.328s
2023-01-11T21:21:28.1256962Z 
2023-01-11T21:21:28.1257091Z OK (skipped=1)
2023-01-11T21:21:28.1257096Z 
2023-01-11T21:21:28.1257173Z Generating XML reports...
2023-01-11T21:21:28.1257782Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212027.xml
2023-01-11T21:21:28.1258071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1258195Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1258478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1258614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1258803Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxeet94pd
2023-01-11T21:21:28.1259003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxeet94pd/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1259274Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1259281Z 
2023-01-11T21:21:28.1259340Z Running tests...
2023-01-11T21:21:28.1259542Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1259822Z   test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5996
2023-01-11T21:21:28.1259980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5997
2023-01-11T21:21:28.1260132Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 5998
2023-01-11T21:21:28.1260282Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 5999
2023-01-11T21:21:28.1260561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1260683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1260965Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1261088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1261273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ci9ja9v
2023-01-11T21:21:28.1261468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ci9ja9v/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1261853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1261977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1262259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1262393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1262578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpydbhowww
2023-01-11T21:21:28.1262759Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpydbhowww/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1263089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1263213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1263493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1263626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1263809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp61zpvpua
2023-01-11T21:21:28.1264005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp61zpvpua/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1264279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1264402Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1264715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1264852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1265036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0usm6umv
2023-01-11T21:21:28.1265228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0usm6umv/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1265392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1265556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1265713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1265873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1265961Z skip: Need at least 1 CUDA device (2.226s)
2023-01-11T21:21:28.1265985Z 
2023-01-11T21:21:28.1266173Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1266248Z Ran 1 test in 2.226s
2023-01-11T21:21:28.1266254Z 
2023-01-11T21:21:28.1266326Z OK (skipped=1)
2023-01-11T21:21:28.1266331Z 
2023-01-11T21:21:28.1266415Z Generating XML reports...
2023-01-11T21:21:28.1266840Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212031.xml
2023-01-11T21:21:28.1267116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1267238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1267514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1267635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1267824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp804qor3a
2023-01-11T21:21:28.1268018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp804qor3a/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1268286Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1268291Z 
2023-01-11T21:21:28.1268363Z Running tests...
2023-01-11T21:21:28.1268560Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1268835Z   test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6051
2023-01-11T21:21:28.1268989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6052
2023-01-11T21:21:28.1269141Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6053
2023-01-11T21:21:28.1269280Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6054
2023-01-11T21:21:28.1269586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1269709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1269990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1270123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1270308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbs5rim2m
2023-01-11T21:21:28.1270504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbs5rim2m/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1270774Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1270884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1271188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1271325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1271510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1zue12g
2023-01-11T21:21:28.1271703Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1zue12g/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1271974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1272096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1272374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1272508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1272679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8br5xe8h
2023-01-11T21:21:28.1272875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8br5xe8h/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1273142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1273266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1273543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1273676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1273860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo334uz9u
2023-01-11T21:21:28.1274051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo334uz9u/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1274204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1274368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1274527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1274686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1274787Z skip: Need at least 1 CUDA device (2.224s)
2023-01-11T21:21:28.1274792Z 
2023-01-11T21:21:28.1274990Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1275067Z Ran 1 test in 2.224s
2023-01-11T21:21:28.1275072Z 
2023-01-11T21:21:28.1275143Z OK (skipped=1)
2023-01-11T21:21:28.1275148Z 
2023-01-11T21:21:28.1275233Z Generating XML reports...
2023-01-11T21:21:28.1275635Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212035.xml
2023-01-11T21:21:28.1275944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1276066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1276345Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1276478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1276663Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvp784m0s
2023-01-11T21:21:28.1276861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvp784m0s/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1277129Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1277134Z 
2023-01-11T21:21:28.1277207Z Running tests...
2023-01-11T21:21:28.1277391Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1277693Z   test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6106
2023-01-11T21:21:28.1277852Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6107
2023-01-11T21:21:28.1278004Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6108
2023-01-11T21:21:28.1278153Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6109
2023-01-11T21:21:28.1278424Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1278545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1278823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1278945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1279134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwhnaekvg
2023-01-11T21:21:28.1279328Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwhnaekvg/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1279601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1279722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1280000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1280136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1280318Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8hc5kh7a
2023-01-11T21:21:28.1280511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8hc5kh7a/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1280769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1280892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1281170Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1281303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1281485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn1tv808n
2023-01-11T21:21:28.1281676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn1tv808n/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1281946Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1282068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1282352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1282505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1282690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe0m5q5a2
2023-01-11T21:21:28.1282880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe0m5q5a2/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1283042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1283203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1283363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1283523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1283624Z skip: Need at least 2 CUDA devices (2.230s)
2023-01-11T21:21:28.1283630Z 
2023-01-11T21:21:28.1283851Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1283938Z Ran 1 test in 2.230s
2023-01-11T21:21:28.1283943Z 
2023-01-11T21:21:28.1284001Z OK (skipped=1)
2023-01-11T21:21:28.1284018Z 
2023-01-11T21:21:28.1284089Z Generating XML reports...
2023-01-11T21:21:28.1284506Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212039.xml
2023-01-11T21:21:28.1284775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1284898Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1285176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1285308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1285493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc2adtw6u
2023-01-11T21:21:28.1285689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc2adtw6u/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1285943Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1285960Z 
2023-01-11T21:21:28.1286020Z Running tests...
2023-01-11T21:21:28.1286214Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1286490Z   test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6161
2023-01-11T21:21:28.1286643Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6162
2023-01-11T21:21:28.1286792Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6163
2023-01-11T21:21:28.1286941Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6164
2023-01-11T21:21:28.1287219Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1287340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1287607Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1287741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1287925Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjnu6m4o1
2023-01-11T21:21:28.1288118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjnu6m4o1/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1288387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1288508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1288791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1288954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1289123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3mt304ut
2023-01-11T21:21:28.1289315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3mt304ut/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1289582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1289702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1289979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1290110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1290292Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnwo7ejsv
2023-01-11T21:21:28.1290516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnwo7ejsv/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1290785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1291012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1291293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1291426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1291608Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsb90yx2z
2023-01-11T21:21:28.1291801Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsb90yx2z/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1291963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1292130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1292291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1292450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1292539Z skip: Need at least 2 CUDA devices (2.326s)
2023-01-11T21:21:28.1292545Z 
2023-01-11T21:21:28.1292744Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1292821Z Ran 1 test in 2.327s
2023-01-11T21:21:28.1292826Z 
2023-01-11T21:21:28.1292897Z OK (skipped=1)
2023-01-11T21:21:28.1292901Z 
2023-01-11T21:21:28.1292985Z Generating XML reports...
2023-01-11T21:21:28.1293403Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212043.xml
2023-01-11T21:21:28.1293680Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1293803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1294068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1294200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1294381Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp31odjoa2
2023-01-11T21:21:28.1294574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp31odjoa2/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1294843Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1294848Z 
2023-01-11T21:21:28.1294919Z Running tests...
2023-01-11T21:21:28.1295115Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1295390Z   test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6216
2023-01-11T21:21:28.1295591Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6217
2023-01-11T21:21:28.1295731Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6218
2023-01-11T21:21:28.1295882Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6219
2023-01-11T21:21:28.1296153Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1296280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1296561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1296695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1296978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprb7k9kg9
2023-01-11T21:21:28.1297177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprb7k9kg9/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1297451Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1297561Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1297840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1297976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1298160Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgmff37x4
2023-01-11T21:21:28.1298354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgmff37x4/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1298626Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1298750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1299028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1299148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1299331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpem63wubz
2023-01-11T21:21:28.1299525Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpem63wubz/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1299795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1299921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1300198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1300336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1300520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl1zedl02
2023-01-11T21:21:28.1300712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl1zedl02/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1300862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1301026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1301186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1301345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1301447Z skip: Need at least 2 CUDA devices (2.325s)
2023-01-11T21:21:28.1301452Z 
2023-01-11T21:21:28.1301729Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1301844Z Ran 1 test in 2.326s
2023-01-11T21:21:28.1301849Z 
2023-01-11T21:21:28.1301920Z OK (skipped=1)
2023-01-11T21:21:28.1301925Z 
2023-01-11T21:21:28.1301995Z Generating XML reports...
2023-01-11T21:21:28.1302417Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212047.xml
2023-01-11T21:21:28.1302691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1302813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1303096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1303234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1303418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6pjnqxyv
2023-01-11T21:21:28.1303645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6pjnqxyv/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1303913Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1303918Z 
2023-01-11T21:21:28.1303977Z Running tests...
2023-01-11T21:21:28.1304171Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1304443Z   test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6271
2023-01-11T21:21:28.1304598Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6272
2023-01-11T21:21:28.1304749Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6273
2023-01-11T21:21:28.1304898Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6274
2023-01-11T21:21:28.1305174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1305295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1305575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1305696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1305878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5d1jla9c
2023-01-11T21:21:28.1306070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5d1jla9c/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1306339Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1306459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1306733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1306871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1307052Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpadenfsvn
2023-01-11T21:21:28.1307231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpadenfsvn/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1307500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1307620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1307898Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1308030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1308212Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt4pskw3c
2023-01-11T21:21:28.1308405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt4pskw3c/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1308703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1308825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1309090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1309220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1309401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm7x2jcw4
2023-01-11T21:21:28.1309591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm7x2jcw4/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1309754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1309955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1310118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1310276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1310363Z skip: Need at least 1 CUDA device (2.331s)
2023-01-11T21:21:28.1310382Z 
2023-01-11T21:21:28.1310567Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1310643Z Ran 1 test in 2.331s
2023-01-11T21:21:28.1310649Z 
2023-01-11T21:21:28.1310720Z OK (skipped=1)
2023-01-11T21:21:28.1310725Z 
2023-01-11T21:21:28.1310811Z Generating XML reports...
2023-01-11T21:21:28.1311230Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212052.xml
2023-01-11T21:21:28.1311505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1311633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1311914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1312035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1312218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjgkx1oen
2023-01-11T21:21:28.1312415Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjgkx1oen/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1312682Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1312688Z 
2023-01-11T21:21:28.1312761Z Running tests...
2023-01-11T21:21:28.1312959Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1313234Z   test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6326
2023-01-11T21:21:28.1313393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6327
2023-01-11T21:21:28.1313546Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6328
2023-01-11T21:21:28.1313684Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6329
2023-01-11T21:21:28.1313959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1314082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1314362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1314496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1314677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvn75bj1
2023-01-11T21:21:28.1314906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvn75bj1/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1315175Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1315282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1315560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1315695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1315878Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ko6gnbi
2023-01-11T21:21:28.1316071Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ko6gnbi/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1316341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1316491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1316773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1316909Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1317077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3e_emtxw
2023-01-11T21:21:28.1317269Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3e_emtxw/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1317538Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1317661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1317939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1318073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1318261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp827sutn2
2023-01-11T21:21:28.1318452Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp827sutn2/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1318601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1318763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1318925Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1319089Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1319192Z skip: Need at least 2 CUDA devices (2.329s)
2023-01-11T21:21:28.1319198Z 
2023-01-11T21:21:28.1319397Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1319473Z Ran 1 test in 2.329s
2023-01-11T21:21:28.1319481Z 
2023-01-11T21:21:28.1319555Z OK (skipped=1)
2023-01-11T21:21:28.1319560Z 
2023-01-11T21:21:28.1319645Z Generating XML reports...
2023-01-11T21:21:28.1320052Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212056.xml
2023-01-11T21:21:28.1320322Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1320445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1320720Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1320854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1321037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1xebwxny
2023-01-11T21:21:28.1321236Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1xebwxny/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1321533Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1321539Z 
2023-01-11T21:21:28.1321615Z Running tests...
2023-01-11T21:21:28.1321796Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1322069Z   test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6381
2023-01-11T21:21:28.1322225Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6382
2023-01-11T21:21:28.1322374Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6383
2023-01-11T21:21:28.1322522Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6384
2023-01-11T21:21:28.1322795Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1322944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1323221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1323343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1323525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkt3vo71c
2023-01-11T21:21:28.1323716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkt3vo71c/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1323984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1324103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1324377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1324510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1324693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1qvo0ffx
2023-01-11T21:21:28.1324882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1qvo0ffx/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1325136Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1325255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1325534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1325665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1325848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ny8u4ej
2023-01-11T21:21:28.1326039Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ny8u4ej/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1326310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1326436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1326711Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1326831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1327014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo7q5zv4b
2023-01-11T21:21:28.1327203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo7q5zv4b/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1327364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1327525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1327688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1327875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1327976Z skip: Need at least 2 CUDA devices (2.333s)
2023-01-11T21:21:28.1327981Z 
2023-01-11T21:21:28.1328165Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1328241Z Ran 1 test in 2.333s
2023-01-11T21:21:28.1328246Z 
2023-01-11T21:21:28.1328316Z OK (skipped=1)
2023-01-11T21:21:28.1328320Z 
2023-01-11T21:21:28.1328402Z Generating XML reports...
2023-01-11T21:21:28.1328818Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212100.xml
2023-01-11T21:21:28.1329087Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1329211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1329517Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1329652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1329824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd7twuepn
2023-01-11T21:21:28.1330018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd7twuepn/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1330284Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1330289Z 
2023-01-11T21:21:28.1330360Z Running tests...
2023-01-11T21:21:28.1330554Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1330948Z   test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6436
2023-01-11T21:21:28.1331109Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6437
2023-01-11T21:21:28.1331261Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6438
2023-01-11T21:21:28.1331411Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6439
2023-01-11T21:21:28.1331673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1331796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1332076Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1332211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1332395Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsr1q62x6
2023-01-11T21:21:28.1332591Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsr1q62x6/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1332864Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1332985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1333249Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1333385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1333569Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmgz9ouxp
2023-01-11T21:21:28.1333763Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmgz9ouxp/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1334031Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1334150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1334479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1334611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1334794Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz84ugbwc
2023-01-11T21:21:28.1334974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz84ugbwc/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1335243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1335365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1335640Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1335774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1335991Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27fek7pv
2023-01-11T21:21:28.1336189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27fek7pv/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1336353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1336503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1336663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1336822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1336925Z skip: Need at least 2 CUDA devices (2.232s)
2023-01-11T21:21:28.1336931Z 
2023-01-11T21:21:28.1337128Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1337202Z Ran 1 test in 2.233s
2023-01-11T21:21:28.1337208Z 
2023-01-11T21:21:28.1337279Z OK (skipped=1)
2023-01-11T21:21:28.1337286Z 
2023-01-11T21:21:28.1337371Z Generating XML reports...
2023-01-11T21:21:28.1337788Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212104.xml
2023-01-11T21:21:28.1338047Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1338170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1338448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1338581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1338763Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp86sb4t24
2023-01-11T21:21:28.1338954Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp86sb4t24/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1339223Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1339228Z 
2023-01-11T21:21:28.1339300Z Running tests...
2023-01-11T21:21:28.1339492Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1339758Z   test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6491
2023-01-11T21:21:28.1339912Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6492
2023-01-11T21:21:28.1340063Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6493
2023-01-11T21:21:28.1340212Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6494
2023-01-11T21:21:28.1340482Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1340605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1340914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1341047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1341218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprabfuwt6
2023-01-11T21:21:28.1341411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprabfuwt6/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1341755Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1341882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1342160Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1342296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1342513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg_z8kdwj
2023-01-11T21:21:28.1342708Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg_z8kdwj/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1342978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1343087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1343365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1343500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1343685Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd49wb76q
2023-01-11T21:21:28.1343877Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd49wb76q/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1344149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1344274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1344552Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1344673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1344854Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsi9uqoaz
2023-01-11T21:21:28.1345045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsi9uqoaz/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1345207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1345369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1345529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1345691Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1345790Z skip: Need at least 1 CUDA device (2.331s)
2023-01-11T21:21:28.1345795Z 
2023-01-11T21:21:28.1345992Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1346053Z Ran 1 test in 2.331s
2023-01-11T21:21:28.1346058Z 
2023-01-11T21:21:28.1346127Z OK (skipped=1)
2023-01-11T21:21:28.1346132Z 
2023-01-11T21:21:28.1346218Z Generating XML reports...
2023-01-11T21:21:28.1346634Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212108.xml
2023-01-11T21:21:28.1346903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1347024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1347303Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1347466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1347651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpylwc00zl
2023-01-11T21:21:28.1347833Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpylwc00zl/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1348097Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1348102Z 
2023-01-11T21:21:28.1348173Z Running tests...
2023-01-11T21:21:28.1348367Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1348633Z   test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6546
2023-01-11T21:21:28.1348813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6547
2023-01-11T21:21:28.1348967Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6548
2023-01-11T21:21:28.1349116Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6549
2023-01-11T21:21:28.1349377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1349497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1349777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1349911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1350096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo2sz37cc
2023-01-11T21:21:28.1350289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo2sz37cc/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1350563Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1350686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1350963Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1351084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1351267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprwof_xz_
2023-01-11T21:21:28.1351459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprwof_xz_/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1351728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1351848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1352128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1352265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1352447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1zbd3623
2023-01-11T21:21:28.1352626Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1zbd3623/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1352893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1353013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1353289Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1353421Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1353604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4siygis_
2023-01-11T21:21:28.1353830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4siygis_/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1353995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1354157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1354305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1354463Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1354562Z skip: Need at least 1 CUDA device (2.336s)
2023-01-11T21:21:28.1354568Z 
2023-01-11T21:21:28.1354764Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1354839Z Ran 1 test in 2.336s
2023-01-11T21:21:28.1354844Z 
2023-01-11T21:21:28.1354914Z OK (skipped=1)
2023-01-11T21:21:28.1354918Z 
2023-01-11T21:21:28.1355001Z Generating XML reports...
2023-01-11T21:21:28.1355469Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212112.xml
2023-01-11T21:21:28.1355745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1355855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1356129Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1356265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1356449Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplsh0n5pk
2023-01-11T21:21:28.1356642Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplsh0n5pk/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1356910Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1356918Z 
2023-01-11T21:21:28.1356990Z Running tests...
2023-01-11T21:21:28.1357185Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1357451Z   test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6601
2023-01-11T21:21:28.1357605Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6602
2023-01-11T21:21:28.1357759Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6603
2023-01-11T21:21:28.1357908Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6604
2023-01-11T21:21:28.1358181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1358303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1358585Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1358720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1358903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmed3nzj7
2023-01-11T21:21:28.1359084Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmed3nzj7/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1359350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1359470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1359747Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1359881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1360065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88qb22cu
2023-01-11T21:21:28.1360287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88qb22cu/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1360557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1360677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1360944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1361075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1361258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3w192lxs
2023-01-11T21:21:28.1361446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3w192lxs/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1361746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1361869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1362148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1362281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1362451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd6ub0sf5
2023-01-11T21:21:28.1362643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd6ub0sf5/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1362803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1362965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1363124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1363283Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1363387Z skip: Need at least 4 CUDA devices (2.229s)
2023-01-11T21:21:28.1363392Z 
2023-01-11T21:21:28.1363588Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1363661Z Ran 1 test in 2.229s
2023-01-11T21:21:28.1363666Z 
2023-01-11T21:21:28.1363724Z OK (skipped=1)
2023-01-11T21:21:28.1363729Z 
2023-01-11T21:21:28.1363814Z Generating XML reports...
2023-01-11T21:21:28.1364245Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20230111212116.xml
2023-01-11T21:21:28.1364518Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1364640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1364921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1365057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1365241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsyj22rhf
2023-01-11T21:21:28.1365421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsyj22rhf/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1365689Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1365694Z 
2023-01-11T21:21:28.1365768Z Running tests...
2023-01-11T21:21:28.1365962Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1366240Z   test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6656
2023-01-11T21:21:28.1366397Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6657
2023-01-11T21:21:28.1366583Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6658
2023-01-11T21:21:28.1366732Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6659
2023-01-11T21:21:28.1367006Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1367116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1367401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1367536Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1367722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp96ppb4xw
2023-01-11T21:21:28.1367916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp96ppb4xw/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1368214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1368340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1368616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1368751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1368922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq_79yb15
2023-01-11T21:21:28.1369116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq_79yb15/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1369386Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1369506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1369787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1369925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1370108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqshryzxk
2023-01-11T21:21:28.1370306Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqshryzxk/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1370560Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1370682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1371082Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1371217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1371401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ukhhvsv
2023-01-11T21:21:28.1371597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ukhhvsv/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1371763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1371931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1372091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1372239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1372343Z skip: Need at least 4 CUDA devices (2.233s)
2023-01-11T21:21:28.1372348Z 
2023-01-11T21:21:28.1372546Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1372626Z Ran 1 test in 2.234s
2023-01-11T21:21:28.1372631Z 
2023-01-11T21:21:28.1372704Z OK (skipped=1)
2023-01-11T21:21:28.1372708Z 
2023-01-11T21:21:28.1372793Z Generating XML reports...
2023-01-11T21:21:28.1373227Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20230111212121.xml
2023-01-11T21:21:28.1373545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1373669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1373934Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1374070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1374256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpre2vrhym
2023-01-11T21:21:28.1374451Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpre2vrhym/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1374719Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T21:21:28.1374727Z 
2023-01-11T21:21:28.1374833Z Running tests...
2023-01-11T21:21:28.1375035Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1375316Z   test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6711
2023-01-11T21:21:28.1375458Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6712
2023-01-11T21:21:28.1375609Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 6713
2023-01-11T21:21:28.1375758Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 6714
2023-01-11T21:21:28.1376035Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1376157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1376441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1376581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1376767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpum27odrr
2023-01-11T21:21:28.1376962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpum27odrr/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1377221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1377342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1377616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1377749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1377929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47_4srj6
2023-01-11T21:21:28.1378124Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47_4srj6/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1378391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1378512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1378776Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1378907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1379088Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb9_f1my_
2023-01-11T21:21:28.1379279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb9_f1my_/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1379548Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:21:28.1379702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:21:28.1379976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:21:28.1380109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:21:28.1380292Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdqs7xss9
2023-01-11T21:21:28.1380471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdqs7xss9/_remote_module_non_scriptable.py
2023-01-11T21:21:28.1380632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:21:28.1380794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:21:28.1380953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:21:28.1381140Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:21:28.1381245Z skip: Need at least 4 CUDA devices (2.237s)
2023-01-11T21:21:28.1381250Z 
2023-01-11T21:21:28.1381448Z ----------------------------------------------------------------------
2023-01-11T21:21:28.1381521Z Ran 1 test in 2.237s
2023-01-11T21:21:28.1381526Z 
2023-01-11T21:21:28.1381669Z OK (skipped=1)
2023-01-11T21:21:28.1381677Z 
2023-01-11T21:21:28.1381749Z Generating XML reports...
2023-01-11T21:21:28.1382182Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20230111212125.xml
2023-01-11T21:21:28.1382187Z 
2023-01-11T21:21:28.1382603Z ##[endgroup]
2023-01-11T21:21:28.1382993Z FINISHED PRINTING LOG FILE of distributed/rpc/cuda/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-cuda-test_tensorpipe_agent_ezxa9svg)
2023-01-11T21:21:28.1382998Z 
2023-01-11T21:21:28.1383179Z Running distributed/test_c10d_gloo ... [2023-01-11 21:21:28.018778]
2023-01-11T21:21:28.1383558Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:21:28.019136]
2023-01-11T21:28:27.6083521Z 
2023-01-11T21:28:27.6083897Z Expand the folded group to see the log file of distributed/test_c10d_gloo
2023-01-11T21:28:27.6084608Z ##[group]PRINTING LOG FILE of distributed/test_c10d_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_gloo_fybruq18)
2023-01-11T21:28:27.6087334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp091ze67d
2023-01-11T21:28:27.6088011Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp091ze67d/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6089897Z <unittest.suite.TestSuite tests=[<__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cpu>, <__main__.CommTest testMethod=test_broadcast_coalesced_gloo_cuda>, <__main__.CommTest testMethod=test_gloo_barrier_device_ids>, <__main__.CommTest testMethod=test_gloo_rank_membership>, <__main__.CommTest testMethod=test_gloo_warn_not_in_group>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_default>, <__main__.CommTest testMethod=test_sequence_num_incremented_gloo_subgroup>, <__main__.CommTest testMethod=test_sequence_num_set_default_pg_gloo>, <__main__.CommTest testMethod=test_sequence_num_set_gloo_new_group>, <__main__.CommTest testMethod=test_tensor_dtype_complex>, <__main__.CommTest testMethod=test_tensor_dtype_mismatch>]>
2023-01-11T21:28:27.6091411Z test_broadcast_coalesced_gloo_cpu (__main__.CommTest)
2023-01-11T21:28:27.6091866Z test_broadcast_coalesced_gloo_cuda (__main__.CommTest)
2023-01-11T21:28:27.6092288Z test_gloo_barrier_device_ids (__main__.CommTest)
2023-01-11T21:28:27.6092676Z test_gloo_rank_membership (__main__.CommTest)
2023-01-11T21:28:27.6093085Z test_gloo_warn_not_in_group (__main__.CommTest)
2023-01-11T21:28:27.6093523Z test_sequence_num_incremented_gloo_default (__main__.CommTest)
2023-01-11T21:28:27.6094337Z test_sequence_num_incremented_gloo_subgroup (__main__.CommTest)
2023-01-11T21:28:27.6094766Z test_sequence_num_set_default_pg_gloo (__main__.CommTest)
2023-01-11T21:28:27.6095224Z test_sequence_num_set_gloo_new_group (__main__.CommTest)
2023-01-11T21:28:27.6095634Z test_tensor_dtype_complex (__main__.CommTest)
2023-01-11T21:28:27.6096049Z test_tensor_dtype_mismatch (__main__.CommTest)
2023-01-11T21:28:27.6097797Z <unittest.suite.TestSuite tests=[<__main__.CompilerTest testMethod=test_allgather_work_wait_cpu>, <__main__.CompilerTest testMethod=test_allgather_work_wait_gpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_cpu>, <__main__.CompilerTest testMethod=test_allreduce_work_wait_gpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_cpu>, <__main__.CompilerTest testMethod=test_broadcast_work_wait_gpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_cpu>, <__main__.CompilerTest testMethod=test_consecutive_comm_work_wait_gpu>, <__main__.CompilerTest testMethod=test_nested_comm_tensor_wrapping>, <__main__.CompilerTest testMethod=test_scatter_work_wait_cpu>, <__main__.CompilerTest testMethod=test_scatter_work_wait_gpu>]>
2023-01-11T21:28:27.6099307Z test_allgather_work_wait_cpu (__main__.CompilerTest)
2023-01-11T21:28:27.6099697Z test_allgather_work_wait_gpu (__main__.CompilerTest)
2023-01-11T21:28:27.6100066Z test_allreduce_work_wait_cpu (__main__.CompilerTest)
2023-01-11T21:28:27.6100423Z test_allreduce_work_wait_gpu (__main__.CompilerTest)
2023-01-11T21:28:27.6100738Z test_broadcast_work_wait_cpu (__main__.CompilerTest)
2023-01-11T21:28:27.6101102Z test_broadcast_work_wait_gpu (__main__.CompilerTest)
2023-01-11T21:28:27.6101468Z test_consecutive_comm_work_wait_cpu (__main__.CompilerTest)
2023-01-11T21:28:27.6101864Z test_consecutive_comm_work_wait_gpu (__main__.CompilerTest)
2023-01-11T21:28:27.6102318Z test_nested_comm_tensor_wrapping (__main__.CompilerTest)
2023-01-11T21:28:27.6102713Z test_scatter_work_wait_cpu (__main__.CompilerTest)
2023-01-11T21:28:27.6103167Z test_scatter_work_wait_gpu (__main__.CompilerTest)
2023-01-11T21:28:27.6109944Z <unittest.suite.TestSuite tests=[<__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_module>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_dynamic_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_once_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_static_graph_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_twice_weight_sharing>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_unused_params_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_False>, <__main__.DistributedDataParallelTest testMethod=test_ddp_checkpointing_weight_sharing_use_reentrant_True>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_cpu>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_future_passing_gpu_gloo>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_register_just_once>, <__main__.DistributedDataParallelTest testMethod=test_ddp_comm_hook_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_init>, <__main__.DistributedDataParallelTest testMethod=test_ddp_invalid_comm_hook_return_type>, <__main__.DistributedDataParallelTest testMethod=test_find_unused_parameters_when_unused_parameters_empty>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_global_local_unused_params_grad_with_static_graph>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_integer_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_1gpu_module_device_ids_torch_device_list>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_2gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_4gpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module>, <__main__.DistributedDataParallelTest testMethod=test_gloo_backend_cpu_module_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output>, <__main__.DistributedDataParallelTest testMethod=test_ignored_output_with_unused_parameters>, <__main__.DistributedDataParallelTest testMethod=test_ignored_sharded_tensor>, <__main__.DistributedDataParallelTest testMethod=test_invalid_powerSGD_state>, <__main__.DistributedDataParallelTest testMethod=test_save_load_checkpoint>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients>, <__main__.DistributedDataParallelTest testMethod=test_sparse_gradients_grad_is_view>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_empty_input>, <__main__.DistributedDataParallelTest testMethod=test_sync_batch_norm_only_empty_input>]>
2023-01-11T21:28:27.6116151Z test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6116681Z test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6117192Z test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6117738Z test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6118349Z test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6118975Z test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6119564Z test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6120111Z test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6120666Z test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6121257Z test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6121881Z test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6122486Z test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6123104Z test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6123679Z test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6124204Z test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6124741Z test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6125256Z test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6125721Z test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6126207Z test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6126638Z test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6127143Z test_global_local_unused_params_grad (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6127606Z test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6128171Z test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6128738Z test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6129299Z test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6129818Z test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6130340Z test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6130956Z test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6131386Z test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6131783Z test_ignored_output (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6132330Z test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6132770Z test_ignored_sharded_tensor (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6133193Z test_invalid_powerSGD_state (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6133603Z test_save_load_checkpoint (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6134054Z test_sparse_gradients (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6134492Z test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6135007Z test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6135536Z test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6137125Z <unittest.suite.TestSuite tests=[<__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_all_to_all_single>, <__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_allgather_coalesced>, <__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_allreduce_coalesced>, <__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_collectives>, <__main__.GlooProcessGroupWithDispatchedCollectivesTests testMethod=test_monitored_barrier>]>
2023-01-11T21:28:27.6138359Z test_all_to_all_single (__main__.GlooProcessGroupWithDispatchedCollectivesTests)
2023-01-11T21:28:27.6138903Z test_allgather_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests)
2023-01-11T21:28:27.6139460Z test_allreduce_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests)
2023-01-11T21:28:27.6140031Z test_collectives (__main__.GlooProcessGroupWithDispatchedCollectivesTests)
2023-01-11T21:28:27.6140573Z test_monitored_barrier (__main__.GlooProcessGroupWithDispatchedCollectivesTests)
2023-01-11T21:28:27.6141019Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:28:27.6146671Z <unittest.suite.TestSuite tests=[<__main__.ProcessGroupGlooTest testMethod=test_allgather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allgather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_cuda_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_basics_using_work_api>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_async>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_basics>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_checks_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_coalesced_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_allreduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_barrier_implies_wait>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_checks>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress>, <__main__.ProcessGroupGlooTest testMethod=test_broadcast_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_empty_tensors>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics>, <__main__.ProcessGroupGlooTest testMethod=test_gather_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_gather_checks>, <__main__.ProcessGroupGlooTest testMethod=test_gather_noncontiguous_input>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress>, <__main__.ProcessGroupGlooTest testMethod=test_gather_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_multi_device_constructor>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_checks>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress>, <__main__.ProcessGroupGlooTest testMethod=test_reduce_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin>, <__main__.ProcessGroupGlooTest testMethod=test_round_robin_create_destroy>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_checks>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress>, <__main__.ProcessGroupGlooTest testMethod=test_scatter_stress_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_send_recv_all_to_all>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_basics_cuda>, <__main__.ProcessGroupGlooTest testMethod=test_sparse_allreduce_checks>]>
2023-01-11T21:28:27.6150723Z test_allgather_basics (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6151006Z test_allgather_basics_cuda (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6151277Z test_allgather_checks (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6151541Z test_allgather_coalesced_async (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6151828Z test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6152126Z test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6152410Z test_allgather_stress (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6152671Z test_allgather_stress_cuda (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6152946Z test_allreduce_basics (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6153216Z test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6153493Z test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6153801Z test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6154085Z test_allreduce_checks (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6154349Z test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6154635Z test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6154923Z test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6155217Z test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6155499Z test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6155771Z test_allreduce_stress (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6156045Z test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6156339Z test_barrier_implies_wait (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6156605Z test_broadcast_basics (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6156876Z test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6157129Z test_broadcast_checks (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6157392Z test_broadcast_stress (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6157661Z test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6157912Z test_empty_tensors (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6158167Z test_gather_basics (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6158429Z test_gather_basics_cuda (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6158691Z test_gather_checks (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6158953Z test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6159258Z test_gather_stress (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6159526Z test_gather_stress_cuda (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6159792Z test_multi_device_constructor (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6160061Z test_reduce_basics (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6160325Z test_reduce_basics_cuda (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6160575Z test_reduce_checks (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6160825Z test_reduce_stress (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6161082Z test_reduce_stress_cuda (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6161339Z test_round_robin (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6161597Z test_round_robin_create_destroy (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6161869Z test_scatter_basics (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6162133Z test_scatter_basics_cuda (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6162382Z test_scatter_checks (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6162644Z test_scatter_stress (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6162909Z test_scatter_stress_cuda (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6163165Z test_send_recv_all_to_all (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6163439Z test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6163728Z test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6164003Z test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest)
2023-01-11T21:28:27.6164651Z <unittest.suite.TestSuite tests=[<__main__.ReducerTest testMethod=test_forward_backward>, <__main__.ReducerTest testMethod=test_forward_backward_optimizer>, <__main__.ReducerTest testMethod=test_forward_backward_unused_parameters>, <__main__.ReducerTest testMethod=test_multi_dtype_multi_bucket>, <__main__.ReducerTest testMethod=test_multi_dtype_single_bucket>, <__main__.ReducerTest testMethod=test_single_dtype_single_bucket>]>
2023-01-11T21:28:27.6165258Z test_forward_backward (__main__.ReducerTest)
2023-01-11T21:28:27.6165506Z test_forward_backward_optimizer (__main__.ReducerTest)
2023-01-11T21:28:27.6165773Z test_forward_backward_unused_parameters (__main__.ReducerTest)
2023-01-11T21:28:27.6166023Z test_multi_dtype_multi_bucket (__main__.ReducerTest)
2023-01-11T21:28:27.6166272Z test_multi_dtype_single_bucket (__main__.ReducerTest)
2023-01-11T21:28:27.6166524Z test_single_dtype_single_bucket (__main__.ReducerTest)
2023-01-11T21:28:27.6166820Z <unittest.suite.TestSuite tests=[<__main__.RendezvousEnvTest testMethod=test_logging_init>]>
2023-01-11T21:28:27.6167118Z test_logging_init (__main__.RendezvousEnvTest)
2023-01-11T21:28:27.6167352Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:28:27.6167657Z <unittest.suite.TestSuite tests=[<__main__.TimeoutTest testMethod=test_default_store_timeout_gloo>]>
2023-01-11T21:28:27.6167952Z test_default_store_timeout_gloo (__main__.TimeoutTest)
2023-01-11T21:28:27.6168472Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6168838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6169256Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6169600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6169941Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9ai57kw
2023-01-11T21:28:27.6170339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9ai57kw/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6170768Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6171129Z 
2023-01-11T21:28:27.6171201Z Running tests...
2023-01-11T21:28:27.6171506Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6171935Z   test_broadcast_coalesced_gloo_cpu (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6776
2023-01-11T21:28:27.6172317Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6777
2023-01-11T21:28:27.6172766Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6173092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6173503Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6173844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6174183Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jydxdbj
2023-01-11T21:28:27.6174581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jydxdbj/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6175051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6175374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6175796Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6176126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6176463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44v_tyqh
2023-01-11T21:28:27.6176853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44v_tyqh/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6177224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6177557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6177912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6178279Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6178755Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6179268Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6179551Z ok (1.863s)
2023-01-11T21:28:27.6179651Z 
2023-01-11T21:28:27.6179851Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6180075Z Ran 1 test in 1.863s
2023-01-11T21:28:27.6180189Z 
2023-01-11T21:28:27.6180251Z OK
2023-01-11T21:28:27.6180340Z 
2023-01-11T21:28:27.6180424Z Generating XML reports...
2023-01-11T21:28:27.6180815Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212130.xml
2023-01-11T21:28:27.6181311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6181679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6182102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6182432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6182775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplm81a6ec
2023-01-11T21:28:27.6183259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplm81a6ec/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6183699Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6183876Z 
2023-01-11T21:28:27.6183948Z Running tests...
2023-01-11T21:28:27.6184247Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6184673Z   test_broadcast_coalesced_gloo_cuda (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6817
2023-01-11T21:28:27.6185044Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6818
2023-01-11T21:28:27.6185493Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6185820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6186242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6186573Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6186914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj2xaxr_h
2023-01-11T21:28:27.6187314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj2xaxr_h/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6187676Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6188134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6188458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6190865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6191198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6191539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps0dqml99
2023-01-11T21:28:27.6191934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps0dqml99/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6192307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6192579Z skip: Need at least 2 CUDA devices (1.861s)
2023-01-11T21:28:27.6192713Z 
2023-01-11T21:28:27.6192920Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6193156Z Ran 1 test in 1.861s
2023-01-11T21:28:27.6193265Z 
2023-01-11T21:28:27.6193324Z OK (skipped=1)
2023-01-11T21:28:27.6193430Z 
2023-01-11T21:28:27.6193513Z Generating XML reports...
2023-01-11T21:28:27.6193915Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212133.xml
2023-01-11T21:28:27.6194404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6194718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6195140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6195483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6195813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9v8q7h1m
2023-01-11T21:28:27.6196264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9v8q7h1m/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6196705Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6196892Z 
2023-01-11T21:28:27.6196963Z Running tests...
2023-01-11T21:28:27.6197247Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6197612Z   test_gloo_barrier_device_ids (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6850
2023-01-11T21:28:27.6197985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6851
2023-01-11T21:28:27.6198433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6198745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6199199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6199542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6199872Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprscnynu0
2023-01-11T21:28:27.6200268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprscnynu0/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6200750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6201072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6201480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6201821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6202157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2fkek32n
2023-01-11T21:28:27.6202538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2fkek32n/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6202909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6203254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6203611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6203963Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6204449Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6204964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6205247Z ok (1.859s)
2023-01-11T21:28:27.6205337Z 
2023-01-11T21:28:27.6205537Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6205774Z Ran 1 test in 1.860s
2023-01-11T21:28:27.6205884Z 
2023-01-11T21:28:27.6205943Z OK
2023-01-11T21:28:27.6206032Z 
2023-01-11T21:28:27.6206103Z Generating XML reports...
2023-01-11T21:28:27.6206503Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212137.xml
2023-01-11T21:28:27.6206998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6207321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6207730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6208070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6208410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ajn23_i
2023-01-11T21:28:27.6208829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ajn23_i/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6209266Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6209457Z 
2023-01-11T21:28:27.6209530Z Running tests...
2023-01-11T21:28:27.6209829Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6210184Z   test_gloo_rank_membership (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6889
2023-01-11T21:28:27.6210552Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6890
2023-01-11T21:28:27.6211155Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6211472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6211960Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6212306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6212646Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehksnlec
2023-01-11T21:28:27.6213031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehksnlec/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6213404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6213856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6214181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6214591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6214928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6215271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphlv_mcem
2023-01-11T21:28:27.6215656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphlv_mcem/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6216028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6216312Z skip: Need at least 2 CUDA devices (1.763s)
2023-01-11T21:28:27.6216447Z 
2023-01-11T21:28:27.6216643Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6216867Z Ran 1 test in 1.763s
2023-01-11T21:28:27.6216977Z 
2023-01-11T21:28:27.6217047Z OK (skipped=1)
2023-01-11T21:28:27.6217152Z 
2023-01-11T21:28:27.6217234Z Generating XML reports...
2023-01-11T21:28:27.6217620Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212140.xml
2023-01-11T21:28:27.6218114Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6218440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6218862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6219188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6219527Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp72vh5qri
2023-01-11T21:28:27.6219917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp72vh5qri/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6220339Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6220526Z 
2023-01-11T21:28:27.6220598Z Running tests...
2023-01-11T21:28:27.6220894Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6221261Z   test_gloo_warn_not_in_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6922
2023-01-11T21:28:27.6221655Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6923
2023-01-11T21:28:27.6222102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6222424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6222917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6223251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6223593Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy2vs2_8p
2023-01-11T21:28:27.6224067Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6224376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6224832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6225172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6225514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9xjps1d2
2023-01-11T21:28:27.6225899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy2vs2_8p/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6226300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xjps1d2/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6226674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6227006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6227290Z skip: Need at least 2 CUDA devices (1.860s)
2023-01-11T21:28:27.6227424Z 
2023-01-11T21:28:27.6227624Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6227859Z Ran 1 test in 1.860s
2023-01-11T21:28:27.6227957Z 
2023-01-11T21:28:27.6228028Z OK (skipped=1)
2023-01-11T21:28:27.6228134Z 
2023-01-11T21:28:27.6228215Z Generating XML reports...
2023-01-11T21:28:27.6228613Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212144.xml
2023-01-11T21:28:27.6229090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6229417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6229835Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6230174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6230500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmc3citxz
2023-01-11T21:28:27.6230902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmc3citxz/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6231342Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6231571Z 
2023-01-11T21:28:27.6254050Z Running tests...
2023-01-11T21:28:27.6254674Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6255295Z   test_sequence_num_incremented_gloo_default (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6955
2023-01-11T21:28:27.6255884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6956
2023-01-11T21:28:27.6256594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6257100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6257800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6258510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6259089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxt2fghot
2023-01-11T21:28:27.6259755Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxt2fghot/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6260612Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6261158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6261873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6262449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6263091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcnatiqmv
2023-01-11T21:28:27.6263858Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcnatiqmv/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6264489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6265075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6265561Z skip: Need at least 2 CUDA devices (1.865s)
2023-01-11T21:28:27.6265788Z 
2023-01-11T21:28:27.6266136Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6266550Z Ran 1 test in 1.865s
2023-01-11T21:28:27.6266739Z 
2023-01-11T21:28:27.6266858Z OK (skipped=1)
2023-01-11T21:28:27.6267043Z 
2023-01-11T21:28:27.6267194Z Generating XML reports...
2023-01-11T21:28:27.6267861Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212147.xml
2023-01-11T21:28:27.6268701Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6269267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6269976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6270557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6271125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjstt0wnv
2023-01-11T21:28:27.6271808Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjstt0wnv/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6272542Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6272867Z 
2023-01-11T21:28:27.6272986Z Running tests...
2023-01-11T21:28:27.6273493Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6274156Z   test_sequence_num_incremented_gloo_subgroup (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6988
2023-01-11T21:28:27.6274805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6989
2023-01-11T21:28:27.6275577Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6276055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6276675Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6277154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6277661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgbb0pffj
2023-01-11T21:28:27.6278116Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgbb0pffj/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6278598Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6279002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6279433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6279777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6280108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph0hyd17n
2023-01-11T21:28:27.6280506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph0hyd17n/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6280883Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6281219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6281502Z skip: Need at least 4 CUDA devices (1.862s)
2023-01-11T21:28:27.6281638Z 
2023-01-11T21:28:27.6281878Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6282122Z Ran 1 test in 1.863s
2023-01-11T21:28:27.6282222Z 
2023-01-11T21:28:27.6282295Z OK (skipped=1)
2023-01-11T21:28:27.6282403Z 
2023-01-11T21:28:27.6282489Z Generating XML reports...
2023-01-11T21:28:27.6282893Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212151.xml
2023-01-11T21:28:27.6283372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6283699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6284126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6284469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6284798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_da8_fd
2023-01-11T21:28:27.6285198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_da8_fd/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6285636Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6285823Z 
2023-01-11T21:28:27.6285882Z Running tests...
2023-01-11T21:28:27.6286173Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6286552Z   test_sequence_num_set_default_pg_gloo (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7021
2023-01-11T21:28:27.6286934Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7022
2023-01-11T21:28:27.6287370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6287699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6288128Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6288476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6288808Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzkk3xpsi
2023-01-11T21:28:27.6289209Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzkk3xpsi/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6289585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6290028Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6290355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6290781Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6291276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6291675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjv83tb47
2023-01-11T21:28:27.6292072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjv83tb47/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6292448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6292722Z skip: Need at least 2 CUDA devices (1.863s)
2023-01-11T21:28:27.6292858Z 
2023-01-11T21:28:27.6293061Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6293299Z Ran 1 test in 1.864s
2023-01-11T21:28:27.6293411Z 
2023-01-11T21:28:27.6293483Z OK (skipped=1)
2023-01-11T21:28:27.6293577Z 
2023-01-11T21:28:27.6293662Z Generating XML reports...
2023-01-11T21:28:27.6294065Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212154.xml
2023-01-11T21:28:27.6294596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6294917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6295342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6295683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6296024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphw_sllhw
2023-01-11T21:28:27.6296409Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphw_sllhw/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6296849Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6297042Z 
2023-01-11T21:28:27.6297114Z Running tests...
2023-01-11T21:28:27.6297401Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6297782Z   test_sequence_num_set_gloo_new_group (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7054
2023-01-11T21:28:27.6298164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7055
2023-01-11T21:28:27.6298610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6298924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6299347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6299689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6300034Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx0yiu5pu
2023-01-11T21:28:27.6300422Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx0yiu5pu/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6300796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6301253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6301571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6301995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6302336Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6302674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78c8boay
2023-01-11T21:28:27.6303130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78c8boay/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6303504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6303794Z skip: Need at least 2 CUDA devices (1.767s)
2023-01-11T21:28:27.6303927Z 
2023-01-11T21:28:27.6304126Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6304409Z Ran 1 test in 1.768s
2023-01-11T21:28:27.6304521Z 
2023-01-11T21:28:27.6304593Z OK (skipped=1)
2023-01-11T21:28:27.6304700Z 
2023-01-11T21:28:27.6304785Z Generating XML reports...
2023-01-11T21:28:27.6305173Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212157.xml
2023-01-11T21:28:27.6305663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6305991Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6306401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6306745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6307087Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo5ywxqy3
2023-01-11T21:28:27.6307520Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo5ywxqy3/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6307950Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6308140Z 
2023-01-11T21:28:27.6308213Z Running tests...
2023-01-11T21:28:27.6308514Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6308881Z   test_tensor_dtype_complex (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7087
2023-01-11T21:28:27.6309240Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7088
2023-01-11T21:28:27.6309687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6310014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6310426Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6310770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6311110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_3ex8d2m
2023-01-11T21:28:27.6311505Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_3ex8d2m/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6311863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6312317Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6312643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6313052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6313394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6313741Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkx7oo91f
2023-01-11T21:28:27.6314138Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkx7oo91f/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6314496Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6314783Z skip: Need at least 2 CUDA devices (1.861s)
2023-01-11T21:28:27.6314917Z 
2023-01-11T21:28:27.6315116Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6315353Z Ran 1 test in 1.861s
2023-01-11T21:28:27.6315452Z 
2023-01-11T21:28:27.6315524Z OK (skipped=1)
2023-01-11T21:28:27.6315629Z 
2023-01-11T21:28:27.6315713Z Generating XML reports...
2023-01-11T21:28:27.6316116Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212201.xml
2023-01-11T21:28:27.6316592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6316953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6317374Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6317714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6318042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqpzb_tn2
2023-01-11T21:28:27.6318443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqpzb_tn2/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6318880Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6319070Z 
2023-01-11T21:28:27.6319128Z Running tests...
2023-01-11T21:28:27.6319430Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6319827Z   test_tensor_dtype_mismatch (__main__.CommTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7120
2023-01-11T21:28:27.6320203Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7121
2023-01-11T21:28:27.6320639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6320965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6321384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6321709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6322057Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz0e6cran
2023-01-11T21:28:27.6322455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz0e6cran/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6322814Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6323269Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6323591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6324014Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6324339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6324681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9zm99ql_
2023-01-11T21:28:27.6325075Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9zm99ql_/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6325432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6325714Z skip: Need at least 2 CUDA devices (1.761s)
2023-01-11T21:28:27.6325848Z 
2023-01-11T21:28:27.6326049Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6326288Z Ran 1 test in 1.762s
2023-01-11T21:28:27.6326398Z 
2023-01-11T21:28:27.6326456Z OK (skipped=1)
2023-01-11T21:28:27.6326560Z 
2023-01-11T21:28:27.6326643Z Generating XML reports...
2023-01-11T21:28:27.6327043Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212204.xml
2023-01-11T21:28:27.6327521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6327844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6328265Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6328607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6328936Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnwmhiwo9
2023-01-11T21:28:27.6329335Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnwmhiwo9/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6329806Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6329993Z 
2023-01-11T21:28:27.6330064Z Running tests...
2023-01-11T21:28:27.6330346Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6330719Z   test_allgather_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7153
2023-01-11T21:28:27.6331245Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7154
2023-01-11T21:28:27.6331684Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6332010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6332495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6332845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6333174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ea1c_ff
2023-01-11T21:28:27.6333571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ea1c_ff/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6334060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6334386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6334794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6335136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6335476Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz5mxhmhw
2023-01-11T21:28:27.6335865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz5mxhmhw/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6336238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6336576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6336932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6337283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6337775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6338288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6338985Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6339521Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6340158Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6340687Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6340925Z ok (1.857s)
2023-01-11T21:28:27.6341014Z 
2023-01-11T21:28:27.6341212Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6341449Z Ran 1 test in 1.857s
2023-01-11T21:28:27.6341558Z 
2023-01-11T21:28:27.6341620Z OK
2023-01-11T21:28:27.6341710Z 
2023-01-11T21:28:27.6341781Z Generating XML reports...
2023-01-11T21:28:27.6342245Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212208.xml
2023-01-11T21:28:27.6342746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6343142Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6343559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6343898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6344240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmkk099zo
2023-01-11T21:28:27.6344624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmkk099zo/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6345065Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6345256Z 
2023-01-11T21:28:27.6345361Z Running tests...
2023-01-11T21:28:27.6345664Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6346025Z   test_allgather_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7192
2023-01-11T21:28:27.6346400Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7193
2023-01-11T21:28:27.6346842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6347168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6347580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6347919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6348258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3ct6_ch
2023-01-11T21:28:27.6348642Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3ct6_ch/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6349127Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6349453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6349880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6350208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6350549Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlksfwgu
2023-01-11T21:28:27.6350946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlksfwgu/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6351309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6351661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6351942Z skip: Need at least 2 CUDA devices (1.860s)
2023-01-11T21:28:27.6352077Z 
2023-01-11T21:28:27.6352271Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6352495Z Ran 1 test in 1.861s
2023-01-11T21:28:27.6352604Z 
2023-01-11T21:28:27.6352676Z OK (skipped=1)
2023-01-11T21:28:27.6352783Z 
2023-01-11T21:28:27.6352868Z Generating XML reports...
2023-01-11T21:28:27.6353269Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212211.xml
2023-01-11T21:28:27.6353771Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6354099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6354527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6354897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6355241Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo3facsu5
2023-01-11T21:28:27.6355640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo3facsu5/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6356081Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6356255Z 
2023-01-11T21:28:27.6356329Z Running tests...
2023-01-11T21:28:27.6356628Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6357007Z   test_allreduce_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7225
2023-01-11T21:28:27.6357374Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7226
2023-01-11T21:28:27.6357853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6358184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6358608Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6358939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6359283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptkbiwcd2
2023-01-11T21:28:27.6359687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptkbiwcd2/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6360154Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6360479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6360902Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6361247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6361575Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx80juhm7
2023-01-11T21:28:27.6361969Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx80juhm7/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6362338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6362677Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6363021Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6363386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6363879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6364387Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6365090Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6365624Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6366269Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6366795Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6367418Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6367986Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6368624Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6369151Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6369372Z ok (1.858s)
2023-01-11T21:28:27.6369472Z 
2023-01-11T21:28:27.6369668Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6369905Z Ran 1 test in 1.859s
2023-01-11T21:28:27.6370016Z 
2023-01-11T21:28:27.6370076Z OK
2023-01-11T21:28:27.6370153Z 
2023-01-11T21:28:27.6370274Z Generating XML reports...
2023-01-11T21:28:27.6370691Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212214.xml
2023-01-11T21:28:27.6371350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6371666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6372094Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6372437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6372784Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplnr2l0oa
2023-01-11T21:28:27.6373171Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplnr2l0oa/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6373615Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6373807Z 
2023-01-11T21:28:27.6373880Z Running tests...
2023-01-11T21:28:27.6374165Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6374536Z   test_allreduce_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7264
2023-01-11T21:28:27.6374913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7265
2023-01-11T21:28:27.6375362Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6375673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6376095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6376437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6376771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyv9qhu_0
2023-01-11T21:28:27.6377244Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6377565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6377986Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6378311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6378650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxezrf4ar
2023-01-11T21:28:27.6379045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyv9qhu_0/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6379447Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxezrf4ar/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6379807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6380227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6380510Z skip: Need at least 2 CUDA devices (1.861s)
2023-01-11T21:28:27.6380644Z 
2023-01-11T21:28:27.6380832Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6381070Z Ran 1 test in 1.861s
2023-01-11T21:28:27.6381181Z 
2023-01-11T21:28:27.6381254Z OK (skipped=1)
2023-01-11T21:28:27.6381361Z 
2023-01-11T21:28:27.6381445Z Generating XML reports...
2023-01-11T21:28:27.6381847Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212218.xml
2023-01-11T21:28:27.6382349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6382675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6383183Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6383532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6383876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsw7qlm0y
2023-01-11T21:28:27.6384273Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsw7qlm0y/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6384703Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6384895Z 
2023-01-11T21:28:27.6384967Z Running tests...
2023-01-11T21:28:27.6385265Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6385628Z   test_broadcast_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7297
2023-01-11T21:28:27.6386007Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7298
2023-01-11T21:28:27.6386457Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6386782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6387193Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6387534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6387874Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqq_4hvi5
2023-01-11T21:28:27.6388267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqq_4hvi5/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6388626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6389080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6389406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6389823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6390162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6390500Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5wv4yut7
2023-01-11T21:28:27.6390894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5wv4yut7/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6391250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6391609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6391973Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6392453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6393021Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6393719Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6394263Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6394905Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6395430Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6395668Z ok (1.859s)
2023-01-11T21:28:27.6395773Z 
2023-01-11T21:28:27.6396002Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6396228Z Ran 1 test in 1.860s
2023-01-11T21:28:27.6396338Z 
2023-01-11T21:28:27.6396399Z OK
2023-01-11T21:28:27.6396492Z 
2023-01-11T21:28:27.6396574Z Generating XML reports...
2023-01-11T21:28:27.6396992Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212221.xml
2023-01-11T21:28:27.6397479Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6397806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6398229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6398572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6398902Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xafn8oe
2023-01-11T21:28:27.6399300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xafn8oe/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6399737Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6399924Z 
2023-01-11T21:28:27.6399983Z Running tests...
2023-01-11T21:28:27.6400279Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6400653Z   test_broadcast_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7336
2023-01-11T21:28:27.6401032Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7337
2023-01-11T21:28:27.6401469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6401793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6402221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6402551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6402892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq6hlwhsr
2023-01-11T21:28:27.6403289Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq6hlwhsr/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6403662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6404102Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6404424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6404850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6405194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6405558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplqdysx2a
2023-01-11T21:28:27.6405958Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplqdysx2a/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6406336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6406615Z skip: Need at least 2 CUDA devices (1.860s)
2023-01-11T21:28:27.6406751Z 
2023-01-11T21:28:27.6406952Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6407192Z Ran 1 test in 1.860s
2023-01-11T21:28:27.6407305Z 
2023-01-11T21:28:27.6407377Z OK (skipped=1)
2023-01-11T21:28:27.6407472Z 
2023-01-11T21:28:27.6407556Z Generating XML reports...
2023-01-11T21:28:27.6407977Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212225.xml
2023-01-11T21:28:27.6408511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6408830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6409257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6409599Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6409946Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlfzi_4f
2023-01-11T21:28:27.6410334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlfzi_4f/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6410779Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6411177Z 
2023-01-11T21:28:27.6411283Z Running tests...
2023-01-11T21:28:27.6411710Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6412167Z   test_consecutive_comm_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7369
2023-01-11T21:28:27.6412559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7370
2023-01-11T21:28:27.6413010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6413325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6413756Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6414105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6414442Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzvffitrv
2023-01-11T21:28:27.6414843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzvffitrv/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6415228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6415686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6415996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6416422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6416763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6417102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj6y9rjiu
2023-01-11T21:28:27.6417485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj6y9rjiu/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6417857Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6418218Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6418634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6419125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6419647Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6420344Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6420882Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6421553Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6422090Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6422724Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6423318Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6423945Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6424478Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6425126Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6425653Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6426271Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6426797Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6427431Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant2 target _tensor_constant2 _tensor_constant2 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6427963Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6428604Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant3 target _tensor_constant3 _tensor_constant3 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6429118Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6429351Z ok (1.968s)
2023-01-11T21:28:27.6429451Z 
2023-01-11T21:28:27.6429648Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6429885Z Ran 1 test in 1.968s
2023-01-11T21:28:27.6429982Z 
2023-01-11T21:28:27.6430043Z OK
2023-01-11T21:28:27.6430133Z 
2023-01-11T21:28:27.6430215Z Generating XML reports...
2023-01-11T21:28:27.6430634Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212228.xml
2023-01-11T21:28:27.6431163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6431488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6431912Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6432254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6432581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps5wd6ysj
2023-01-11T21:28:27.6432980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps5wd6ysj/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6433419Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6433607Z 
2023-01-11T21:28:27.6433666Z Running tests...
2023-01-11T21:28:27.6433996Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6434380Z   test_consecutive_comm_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7408
2023-01-11T21:28:27.6434763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7409
2023-01-11T21:28:27.6435199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6435522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6435948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6436278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6436618Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmf4lh2aj
2023-01-11T21:28:27.6437019Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmf4lh2aj/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6437395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6437841Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6438162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6438583Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6438922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6439249Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe_w39sh9
2023-01-11T21:28:27.6439640Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe_w39sh9/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6440008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6440284Z skip: Need at least 2 CUDA devices (1.867s)
2023-01-11T21:28:27.6440419Z 
2023-01-11T21:28:27.6440614Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6440849Z Ran 1 test in 1.867s
2023-01-11T21:28:27.6440958Z 
2023-01-11T21:28:27.6441028Z OK (skipped=1)
2023-01-11T21:28:27.6441122Z 
2023-01-11T21:28:27.6441205Z Generating XML reports...
2023-01-11T21:28:27.6441617Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212232.xml
2023-01-11T21:28:27.6442119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6442431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6442853Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6443191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6443565Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnr3qs7sb
2023-01-11T21:28:27.6443948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnr3qs7sb/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6444389Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6444576Z 
2023-01-11T21:28:27.6444647Z Running tests...
2023-01-11T21:28:27.6444933Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6445310Z   test_nested_comm_tensor_wrapping (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7441
2023-01-11T21:28:27.6445689Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7442
2023-01-11T21:28:27.6446133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6446474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6446906Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6447247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6447587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnx6lxdr2
2023-01-11T21:28:27.6447975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnx6lxdr2/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6448349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6448804Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6449118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6449543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6449884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6450225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42d4uos7
2023-01-11T21:28:27.6450605Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42d4uos7/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6451131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6451493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6451851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6452345Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6452866Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6453569Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6454107Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6454736Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6455265Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6455907Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6456490Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6457113Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant1 target _tensor_constant1 _tensor_constant1 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6457640Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6457879Z ok (1.967s)
2023-01-11T21:28:27.6457984Z 
2023-01-11T21:28:27.6458185Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6458411Z Ran 1 test in 1.968s
2023-01-11T21:28:27.6458524Z 
2023-01-11T21:28:27.6458585Z OK
2023-01-11T21:28:27.6458674Z 
2023-01-11T21:28:27.6458756Z Generating XML reports...
2023-01-11T21:28:27.6459195Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212235.xml
2023-01-11T21:28:27.6459703Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6460026Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6460453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6460781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6461118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppqwvlcxd
2023-01-11T21:28:27.6461522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppqwvlcxd/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6461965Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6462142Z 
2023-01-11T21:28:27.6462213Z Running tests...
2023-01-11T21:28:27.6462516Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6462958Z   test_scatter_work_wait_cpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7480
2023-01-11T21:28:27.6463324Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7481
2023-01-11T21:28:27.6463775Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6464102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6464535Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6464870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6465213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8k3wlvvn
2023-01-11T21:28:27.6465616Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8k3wlvvn/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6465978Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6466439Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6466765Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6467192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6467523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6467870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe8pqx6jl
2023-01-11T21:28:27.6468270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe8pqx6jl/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6468649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6469033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6469400Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6469894Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6470400Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6471098Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6471631Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6472309Z /opt/conda/lib/python3.7/site-packages/torch/fx/graph.py:1346: UserWarning: Node _tensor_constant0 target _tensor_constant0 _tensor_constant0 of  does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target
2023-01-11T21:28:27.6472842Z   warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '
2023-01-11T21:28:27.6473066Z ok (1.960s)
2023-01-11T21:28:27.6473171Z 
2023-01-11T21:28:27.6473369Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6473611Z Ran 1 test in 1.961s
2023-01-11T21:28:27.6473721Z 
2023-01-11T21:28:27.6473771Z OK
2023-01-11T21:28:27.6473862Z 
2023-01-11T21:28:27.6473947Z Generating XML reports...
2023-01-11T21:28:27.6474365Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212239.xml
2023-01-11T21:28:27.6474872Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6475193Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6475613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6475953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6476284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5scnxkbe
2023-01-11T21:28:27.6476682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5scnxkbe/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6477118Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6477305Z 
2023-01-11T21:28:27.6477377Z Running tests...
2023-01-11T21:28:27.6477662Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6478034Z   test_scatter_work_wait_gpu (__main__.CompilerTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7519
2023-01-11T21:28:27.6478411Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7520
2023-01-11T21:28:27.6478842Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6479168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6479591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6479932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6480258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl6qip82n
2023-01-11T21:28:27.6480657Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl6qip82n/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6481028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6481484Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6481845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6482271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6482607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6482930Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyxi0ue9e
2023-01-11T21:28:27.6483325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyxi0ue9e/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6483696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6483987Z skip: Need at least 2 CUDA devices (1.775s)
2023-01-11T21:28:27.6484108Z 
2023-01-11T21:28:27.6484305Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6484577Z Ran 1 test in 1.775s
2023-01-11T21:28:27.6484690Z 
2023-01-11T21:28:27.6484760Z OK (skipped=1)
2023-01-11T21:28:27.6484865Z 
2023-01-11T21:28:27.6484935Z Generating XML reports...
2023-01-11T21:28:27.6485349Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212242.xml
2023-01-11T21:28:27.6485849Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6486172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6486580Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6486920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6487260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx6ct8ivh
2023-01-11T21:28:27.6487647Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx6ct8ivh/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6488090Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6488278Z 
2023-01-11T21:28:27.6488347Z Running tests...
2023-01-11T21:28:27.6488644Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6488956Z   test_ddp_checkpointing_dynamic_module (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6489481Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7552
2023-01-11T21:28:27.6489879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7553
2023-01-11T21:28:27.6490329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6490642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6491228Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6491571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6491898Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjps0ione
2023-01-11T21:28:27.6492304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjps0ione/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6492681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6493138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6493452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6493876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6494221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6494607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5t5o74p0
2023-01-11T21:28:27.6495004Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5t5o74p0/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6495378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6495668Z skip: Need at least 2 CUDA devices (1.866s)
2023-01-11T21:28:27.6495803Z 
2023-01-11T21:28:27.6495990Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6496233Z Ran 1 test in 1.867s
2023-01-11T21:28:27.6496345Z 
2023-01-11T21:28:27.6496417Z OK (skipped=1)
2023-01-11T21:28:27.6496525Z 
2023-01-11T21:28:27.6496596Z Generating XML reports...
2023-01-11T21:28:27.6497059Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212246.xml
2023-01-11T21:28:27.6497643Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6497971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6498385Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6498726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6499068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7l60s9d
2023-01-11T21:28:27.6499469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7l60s9d/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6499897Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6500087Z 
2023-01-11T21:28:27.6500159Z Running tests...
2023-01-11T21:28:27.6500456Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6500784Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6501211Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7585
2023-01-11T21:28:27.6501608Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7586
2023-01-11T21:28:27.6502057Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6502373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6502797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6503214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6503544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxedovvdg
2023-01-11T21:28:27.6503948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxedovvdg/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6504324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6504791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6505103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6505527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6505864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6506206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk9ov3hp4
2023-01-11T21:28:27.6506590Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk9ov3hp4/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6506965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6507286Z skip: Need at least 2 CUDA devices (1.860s)
2023-01-11T21:28:27.6507420Z 
2023-01-11T21:28:27.6507605Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6507842Z Ran 1 test in 1.861s
2023-01-11T21:28:27.6507954Z 
2023-01-11T21:28:27.6508024Z OK (skipped=1)
2023-01-11T21:28:27.6508130Z 
2023-01-11T21:28:27.6508214Z Generating XML reports...
2023-01-11T21:28:27.6508673Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212249.xml
2023-01-11T21:28:27.6509216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6509542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6509954Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6510326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6510668Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp896k5e9o
2023-01-11T21:28:27.6511062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp896k5e9o/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6511485Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6511672Z 
2023-01-11T21:28:27.6511745Z Running tests...
2023-01-11T21:28:27.6512042Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6512364Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6512781Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7618
2023-01-11T21:28:27.6513165Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7619
2023-01-11T21:28:27.6513616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6513928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6514349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6514687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6515031Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkfs0tu9z
2023-01-11T21:28:27.6515416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkfs0tu9z/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6515791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6516242Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6516557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6516982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6517321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6517660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphr9dde99
2023-01-11T21:28:27.6518047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphr9dde99/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6518420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6518708Z skip: Need at least 2 CUDA devices (1.865s)
2023-01-11T21:28:27.6518843Z 
2023-01-11T21:28:27.6519026Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6519268Z Ran 1 test in 1.865s
2023-01-11T21:28:27.6519379Z 
2023-01-11T21:28:27.6519451Z OK (skipped=1)
2023-01-11T21:28:27.6519595Z 
2023-01-11T21:28:27.6519681Z Generating XML reports...
2023-01-11T21:28:27.6520136Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212253.xml
2023-01-11T21:28:27.6520681Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6521010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6521423Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6521766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6522111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8m3o7nh1
2023-01-11T21:28:27.6522511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8m3o7nh1/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6522974Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6523169Z 
2023-01-11T21:28:27.6523242Z Running tests...
2023-01-11T21:28:27.6523544Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6523880Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6524285Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7651
2023-01-11T21:28:27.6524672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7652
2023-01-11T21:28:27.6525122Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6525437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6525867Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6526215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6526560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppp6ktlb5
2023-01-11T21:28:27.6526946Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppp6ktlb5/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6527325Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6527779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6528107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6528519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6528862Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6529206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps3p3kpzl
2023-01-11T21:28:27.6529594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps3p3kpzl/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6529968Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6530251Z skip: Need at least 2 CUDA devices (1.863s)
2023-01-11T21:28:27.6530383Z 
2023-01-11T21:28:27.6530579Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6530926Z Ran 1 test in 1.863s
2023-01-11T21:28:27.6531056Z 
2023-01-11T21:28:27.6531129Z OK (skipped=1)
2023-01-11T21:28:27.6531236Z 
2023-01-11T21:28:27.6531320Z Generating XML reports...
2023-01-11T21:28:27.6531773Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212256.xml
2023-01-11T21:28:27.6532313Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6532729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6533010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6533144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6533315Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn9k0t8kz
2023-01-11T21:28:27.6533511Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn9k0t8kz/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6533741Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6533747Z 
2023-01-11T21:28:27.6533822Z Running tests...
2023-01-11T21:28:27.6534017Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6534208Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6534585Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7684
2023-01-11T21:28:27.6534742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7685
2023-01-11T21:28:27.6535017Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6535126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6535406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6535538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6535722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwdzz3d8f
2023-01-11T21:28:27.6535915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwdzz3d8f/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6536187Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6536308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6536586Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6536719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6536889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_nonnmjq
2023-01-11T21:28:27.6537080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_nonnmjq/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6537244Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6537407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6537512Z skip: Need at least 2 CUDA devices (1.866s)
2023-01-11T21:28:27.6537520Z 
2023-01-11T21:28:27.6537714Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6537789Z Ran 1 test in 1.866s
2023-01-11T21:28:27.6537795Z 
2023-01-11T21:28:27.6537867Z OK (skipped=1)
2023-01-11T21:28:27.6537872Z 
2023-01-11T21:28:27.6537942Z Generating XML reports...
2023-01-11T21:28:27.6538293Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212259.xml
2023-01-11T21:28:27.6538561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6538684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6538962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6539095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6539313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2o0kpogb
2023-01-11T21:28:27.6539507Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2o0kpogb/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6539736Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6539741Z 
2023-01-11T21:28:27.6539801Z Running tests...
2023-01-11T21:28:27.6539991Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6540180Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6540506Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7717
2023-01-11T21:28:27.6540659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7718
2023-01-11T21:28:27.6540959Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6541084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6541405Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6541538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6541709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaxwh29ga
2023-01-11T21:28:27.6541905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaxwh29ga/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6542065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6542333Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6542455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6542736Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6542949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6543135Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzfcvgbds
2023-01-11T21:28:27.6543315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzfcvgbds/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6543476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6543580Z skip: Need at least 2 CUDA devices (1.762s)
2023-01-11T21:28:27.6543585Z 
2023-01-11T21:28:27.6543785Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6543861Z Ran 1 test in 1.762s
2023-01-11T21:28:27.6543866Z 
2023-01-11T21:28:27.6543937Z OK (skipped=1)
2023-01-11T21:28:27.6543942Z 
2023-01-11T21:28:27.6544028Z Generating XML reports...
2023-01-11T21:28:27.6544379Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212303.xml
2023-01-11T21:28:27.6544652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6544761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6545042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6545177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6545363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr78lb806
2023-01-11T21:28:27.6545557Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr78lb806/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6545784Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6545824Z 
2023-01-11T21:28:27.6545897Z Running tests...
2023-01-11T21:28:27.6546091Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6546252Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6546596Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7750
2023-01-11T21:28:27.6546751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7751
2023-01-11T21:28:27.6547025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6547147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6547428Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6547588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6547775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaskpb6kn
2023-01-11T21:28:27.6547968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaskpb6kn/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6548118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6548387Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6548506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6548780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6548912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6549096Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu78y7sbz
2023-01-11T21:28:27.6549293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu78y7sbz/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6549453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6549544Z skip: Need at least 2 CUDA devices (1.758s)
2023-01-11T21:28:27.6549561Z 
2023-01-11T21:28:27.6549742Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6549817Z Ran 1 test in 1.759s
2023-01-11T21:28:27.6549822Z 
2023-01-11T21:28:27.6549891Z OK (skipped=1)
2023-01-11T21:28:27.6549897Z 
2023-01-11T21:28:27.6549980Z Generating XML reports...
2023-01-11T21:28:27.6550324Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212306.xml
2023-01-11T21:28:27.6550592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6550716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6550994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6551115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6551298Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj69s21x8
2023-01-11T21:28:27.6551489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj69s21x8/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6551714Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6551720Z 
2023-01-11T21:28:27.6551791Z Running tests...
2023-01-11T21:28:27.6551983Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6552155Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6552504Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7783
2023-01-11T21:28:27.6552705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7784
2023-01-11T21:28:27.6552962Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6553085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6553365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6553496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6553682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1pcz81h2
2023-01-11T21:28:27.6553875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1pcz81h2/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6554172Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6554296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6554562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6554695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6554876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzf4_5_bx
2023-01-11T21:28:27.6555067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzf4_5_bx/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6555229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6555389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6555493Z skip: Need at least 2 CUDA devices (1.759s)
2023-01-11T21:28:27.6555500Z 
2023-01-11T21:28:27.6555698Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6555773Z Ran 1 test in 1.759s
2023-01-11T21:28:27.6555778Z 
2023-01-11T21:28:27.6555836Z OK (skipped=1)
2023-01-11T21:28:27.6555840Z 
2023-01-11T21:28:27.6555923Z Generating XML reports...
2023-01-11T21:28:27.6556270Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212310.xml
2023-01-11T21:28:27.6556542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6556663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6556940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6557073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6557259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmo00c1_v
2023-01-11T21:28:27.6557456Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmo00c1_v/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6557672Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6557677Z 
2023-01-11T21:28:27.6557748Z Running tests...
2023-01-11T21:28:27.6557941Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6558114Z   test_ddp_checkpointing_twice_weight_sharing (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6558349Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7816
2023-01-11T21:28:27.6558502Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7817
2023-01-11T21:28:27.6558772Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6558925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6559192Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6559325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6559510Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpifpgqzss
2023-01-11T21:28:27.6559705Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpifpgqzss/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6559978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6560099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6560375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6560537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6560723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0g_p27x3
2023-01-11T21:28:27.6560902Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0g_p27x3/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6561068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6561227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6561330Z skip: Need at least 2 CUDA devices (1.861s)
2023-01-11T21:28:27.6561335Z 
2023-01-11T21:28:27.6561533Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6561608Z Ran 1 test in 1.861s
2023-01-11T21:28:27.6561613Z 
2023-01-11T21:28:27.6561683Z OK (skipped=1)
2023-01-11T21:28:27.6561688Z 
2023-01-11T21:28:27.6561771Z Generating XML reports...
2023-01-11T21:28:27.6562105Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212313.xml
2023-01-11T21:28:27.6562379Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6562500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6562777Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6562910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6563091Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzg6twhvf
2023-01-11T21:28:27.6563285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzg6twhvf/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6563513Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6563518Z 
2023-01-11T21:28:27.6563589Z Running tests...
2023-01-11T21:28:27.6563774Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6563960Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6564197Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7849
2023-01-11T21:28:27.6564350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7850
2023-01-11T21:28:27.6564618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6564740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6565019Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6565157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6565376Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx4s3rr_b
2023-01-11T21:28:27.6565558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx4s3rr_b/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6565720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6565990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6566111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6566388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6566521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6566705Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe0eux7hd
2023-01-11T21:28:27.6566928Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe0eux7hd/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6567079Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6567181Z skip: Need at least 2 CUDA devices (1.862s)
2023-01-11T21:28:27.6567186Z 
2023-01-11T21:28:27.6567380Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6567454Z Ran 1 test in 1.862s
2023-01-11T21:28:27.6567459Z 
2023-01-11T21:28:27.6567530Z OK (skipped=1)
2023-01-11T21:28:27.6567534Z 
2023-01-11T21:28:27.6567619Z Generating XML reports...
2023-01-11T21:28:27.6567961Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212316.xml
2023-01-11T21:28:27.6568231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6568352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6568619Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6568750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6568934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnfchn0ef
2023-01-11T21:28:27.6569128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnfchn0ef/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6569357Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6569362Z 
2023-01-11T21:28:27.6569434Z Running tests...
2023-01-11T21:28:27.6569626Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6569809Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6570034Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7882
2023-01-11T21:28:27.6570188Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7883
2023-01-11T21:28:27.6570461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6570582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6571000Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6571138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6571322Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkx5jnbk7
2023-01-11T21:28:27.6571516Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkx5jnbk7/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6571680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6572004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6572125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6572404Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6572541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6572726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5164zd_l
2023-01-11T21:28:27.6572922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5164zd_l/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6573087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6573190Z skip: Need at least 2 CUDA devices (1.761s)
2023-01-11T21:28:27.6573196Z 
2023-01-11T21:28:27.6573430Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6573497Z Ran 1 test in 1.761s
2023-01-11T21:28:27.6573502Z 
2023-01-11T21:28:27.6573574Z OK (skipped=1)
2023-01-11T21:28:27.6573579Z 
2023-01-11T21:28:27.6573664Z Generating XML reports...
2023-01-11T21:28:27.6574012Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212320.xml
2023-01-11T21:28:27.6574284Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6574407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6574687Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6574821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6574994Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwnb5u_2
2023-01-11T21:28:27.6575194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwnb5u_2/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6575425Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6575430Z 
2023-01-11T21:28:27.6575503Z Running tests...
2023-01-11T21:28:27.6575698Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6575890Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6576104Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7915
2023-01-11T21:28:27.6576261Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7916
2023-01-11T21:28:27.6576530Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6576642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6576920Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6577056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6577239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjzoflh6g
2023-01-11T21:28:27.6577432Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjzoflh6g/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6577599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6577871Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6577995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6578272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6578424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6578609Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2eiid8d0
2023-01-11T21:28:27.6578804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2eiid8d0/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6578966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6579075Z skip: Need at least 2 CUDA devices (1.862s)
2023-01-11T21:28:27.6579081Z 
2023-01-11T21:28:27.6579278Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6579353Z Ran 1 test in 1.862s
2023-01-11T21:28:27.6579358Z 
2023-01-11T21:28:27.6579432Z OK (skipped=1)
2023-01-11T21:28:27.6579437Z 
2023-01-11T21:28:27.6579508Z Generating XML reports...
2023-01-11T21:28:27.6579880Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212323.xml
2023-01-11T21:28:27.6580156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6580276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6580557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6580692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6580880Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp3o_8_yi
2023-01-11T21:28:27.6581074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp3o_8_yi/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6581304Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6581309Z 
2023-01-11T21:28:27.6581369Z Running tests...
2023-01-11T21:28:27.6581568Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6581757Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6581968Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7948
2023-01-11T21:28:27.6582123Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7949
2023-01-11T21:28:27.6582394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6582519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6582794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6582968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6583156Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxd1rniy4
2023-01-11T21:28:27.6583351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxd1rniy4/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6583514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6583785Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6583907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6584184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6584318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6584502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnb8lq4gp
2023-01-11T21:28:27.6584685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnb8lq4gp/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6584884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6584988Z skip: Need at least 2 CUDA devices (1.868s)
2023-01-11T21:28:27.6584993Z 
2023-01-11T21:28:27.6585189Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6585264Z Ran 1 test in 1.868s
2023-01-11T21:28:27.6585269Z 
2023-01-11T21:28:27.6585340Z OK (skipped=1)
2023-01-11T21:28:27.6585345Z 
2023-01-11T21:28:27.6585429Z Generating XML reports...
2023-01-11T21:28:27.6585779Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212326.xml
2023-01-11T21:28:27.6586050Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6586158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6586469Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6586606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6586790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz8uhee6c
2023-01-11T21:28:27.6586984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz8uhee6c/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6587211Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6587216Z 
2023-01-11T21:28:27.6587288Z Running tests...
2023-01-11T21:28:27.6587482Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6587630Z   test_ddp_comm_hook_future_passing_cpu (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6587860Z This unit test verifies whether the Future object is passed properly. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7981
2023-01-11T21:28:27.6588017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7982
2023-01-11T21:28:27.6588293Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6588415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6588695Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6588830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6589016Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg8py_twp
2023-01-11T21:28:27.6589212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg8py_twp/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6589361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6589634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6589758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6590037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6590168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6590350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq0ovk53w
2023-01-11T21:28:27.6590543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq0ovk53w/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6590703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6590864Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6591038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6591370Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6591666Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6591732Z ok (1.860s)
2023-01-11T21:28:27.6591738Z 
2023-01-11T21:28:27.6591934Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6592008Z Ran 1 test in 1.861s
2023-01-11T21:28:27.6592013Z 
2023-01-11T21:28:27.6592072Z OK
2023-01-11T21:28:27.6592077Z 
2023-01-11T21:28:27.6592159Z Generating XML reports...
2023-01-11T21:28:27.6592493Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212330.xml
2023-01-11T21:28:27.6592762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6592913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6593196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6593330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6593516Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyp3nk8l6
2023-01-11T21:28:27.6593714Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyp3nk8l6/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6593944Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6593951Z 
2023-01-11T21:28:27.6594022Z Running tests...
2023-01-11T21:28:27.6594203Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6594370Z   test_ddp_comm_hook_future_passing_gpu_gloo (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6594625Z This unit test verifies whether the Future object is passed properly using gloo backend. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8020
2023-01-11T21:28:27.6594780Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8021
2023-01-11T21:28:27.6595052Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6595174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6595453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6595588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6595760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj08wqi7u
2023-01-11T21:28:27.6595953Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj08wqi7u/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6596118Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6596391Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6596514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6596794Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6596927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6597110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp10k86fjt
2023-01-11T21:28:27.6597300Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp10k86fjt/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6597448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6597551Z skip: Need at least 2 CUDA devices (1.864s)
2023-01-11T21:28:27.6597608Z 
2023-01-11T21:28:27.6597806Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6597881Z Ran 1 test in 1.864s
2023-01-11T21:28:27.6597886Z 
2023-01-11T21:28:27.6597958Z OK (skipped=1)
2023-01-11T21:28:27.6597963Z 
2023-01-11T21:28:27.6598046Z Generating XML reports...
2023-01-11T21:28:27.6598392Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212333.xml
2023-01-11T21:28:27.6598663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6598783Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6599051Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6599184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6599400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgwunn1vz
2023-01-11T21:28:27.6599597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgwunn1vz/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6599826Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6599831Z 
2023-01-11T21:28:27.6599904Z Running tests...
2023-01-11T21:28:27.6600100Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6600260Z   test_ddp_comm_hook_register_just_once (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6600494Z DDP communication hook can only be registered once. This test validates whether ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8053
2023-01-11T21:28:27.6600649Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8054
2023-01-11T21:28:27.6600918Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6601045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6601330Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6601463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6601647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe4u9aeia
2023-01-11T21:28:27.6601842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe4u9aeia/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6602003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6602262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6602383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6602666Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6602800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6602984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp316iol8r
2023-01-11T21:28:27.6603176Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp316iol8r/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6603337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6603513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6603673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6603969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6604265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6604362Z ok (1.865s)
2023-01-11T21:28:27.6604367Z 
2023-01-11T21:28:27.6604564Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6604638Z Ran 1 test in 1.865s
2023-01-11T21:28:27.6604643Z 
2023-01-11T21:28:27.6604704Z OK
2023-01-11T21:28:27.6604708Z 
2023-01-11T21:28:27.6604790Z Generating XML reports...
2023-01-11T21:28:27.6605137Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212337.xml
2023-01-11T21:28:27.6605398Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6605519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6605800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6605966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6606152Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgp0uvrvq
2023-01-11T21:28:27.6606347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgp0uvrvq/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6606574Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6606579Z 
2023-01-11T21:28:27.6606649Z Running tests...
2023-01-11T21:28:27.6606844Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6606990Z   test_ddp_comm_hook_sparse_gradients (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6607228Z Runs "test_sparse_gradients" unit test with DDP communication hook. We define a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8092
2023-01-11T21:28:27.6607382Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8093
2023-01-11T21:28:27.6607656Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6607777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6608055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6608187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6608372Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi6uz0y1b
2023-01-11T21:28:27.6608555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi6uz0y1b/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6608825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6608947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6609225Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6609359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6609546Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdmk0sgpz
2023-01-11T21:28:27.6609740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdmk0sgpz/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6609901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6610064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6610228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6610402Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6610702Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6611186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6611258Z ok (1.864s)
2023-01-11T21:28:27.6611263Z 
2023-01-11T21:28:27.6611465Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6611540Z Ran 1 test in 1.864s
2023-01-11T21:28:27.6611545Z 
2023-01-11T21:28:27.6611606Z OK
2023-01-11T21:28:27.6611611Z 
2023-01-11T21:28:27.6611695Z Generating XML reports...
2023-01-11T21:28:27.6612029Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212340.xml
2023-01-11T21:28:27.6612305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6612428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6612763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6612899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6613089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_5cj61kp
2023-01-11T21:28:27.6613282Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_5cj61kp/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6613510Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6613515Z 
2023-01-11T21:28:27.6613574Z Running tests...
2023-01-11T21:28:27.6613773Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6613927Z   test_ddp_invalid_comm_hook_init (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6614166Z This unit test makes sure that register_comm_hook properly checks the format ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8137
2023-01-11T21:28:27.6614322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8138
2023-01-11T21:28:27.6614593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6614719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6615005Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6615139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6615309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_4bjpzrj
2023-01-11T21:28:27.6615506Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_4bjpzrj/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6615773Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6615901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6616177Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6616310Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6616497Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppmc9rs8u
2023-01-11T21:28:27.6616691Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppmc9rs8u/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6616841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6617004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6617178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6617354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6617690Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6617985Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6618051Z ok (1.865s)
2023-01-11T21:28:27.6618056Z 
2023-01-11T21:28:27.6618252Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6618326Z Ran 1 test in 1.865s
2023-01-11T21:28:27.6618331Z 
2023-01-11T21:28:27.6618379Z OK
2023-01-11T21:28:27.6618384Z 
2023-01-11T21:28:27.6618466Z Generating XML reports...
2023-01-11T21:28:27.6618813Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212344.xml
2023-01-11T21:28:27.6619116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6619241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6619521Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6619655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6619837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0lj94xk2
2023-01-11T21:28:27.6620031Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0lj94xk2/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6620245Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6620251Z 
2023-01-11T21:28:27.6620321Z Running tests...
2023-01-11T21:28:27.6620518Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6620678Z   test_ddp_invalid_comm_hook_return_type (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6620924Z This test checks whether return annotation checked properly if defined. It also ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8176
2023-01-11T21:28:27.6621078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8177
2023-01-11T21:28:27.6621348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6621469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6621733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6621865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6622049Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_4_5kfq
2023-01-11T21:28:27.6622244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_4_5kfq/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6622515Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6622637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6622984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6623120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6623304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpowvhw1fn
2023-01-11T21:28:27.6623485Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpowvhw1fn/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6623645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6623807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6624022Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6624198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6624497Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6624793Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6624862Z ok (1.863s)
2023-01-11T21:28:27.6624867Z 
2023-01-11T21:28:27.6625063Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6625125Z Ran 1 test in 1.863s
2023-01-11T21:28:27.6625131Z 
2023-01-11T21:28:27.6625190Z OK
2023-01-11T21:28:27.6625195Z 
2023-01-11T21:28:27.6625277Z Generating XML reports...
2023-01-11T21:28:27.6625648Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212347.xml
2023-01-11T21:28:27.6625923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6626045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6626325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6626457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6626630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiu1xmcn0
2023-01-11T21:28:27.6626827Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiu1xmcn0/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6627055Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6627061Z 
2023-01-11T21:28:27.6627134Z Running tests...
2023-01-11T21:28:27.6627333Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6627523Z   test_find_unused_parameters_when_unused_parameters_empty (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6627758Z An empty unused_parameters array does not imply find_unused_parameters = ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8215
2023-01-11T21:28:27.6627914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8216
2023-01-11T21:28:27.6628188Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6628299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6628582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6628717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6628906Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp75718pfg
2023-01-11T21:28:27.6629104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp75718pfg/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6629268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6629541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6629664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6629943Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6630065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6630251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp53rlh0z4
2023-01-11T21:28:27.6630445Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp53rlh0z4/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6630639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6630741Z skip: Need at least 2 CUDA devices (1.860s)
2023-01-11T21:28:27.6630746Z 
2023-01-11T21:28:27.6630942Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6631020Z Ran 1 test in 1.861s
2023-01-11T21:28:27.6631025Z 
2023-01-11T21:28:27.6631099Z OK (skipped=1)
2023-01-11T21:28:27.6631103Z 
2023-01-11T21:28:27.6631174Z Generating XML reports...
2023-01-11T21:28:27.6631526Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212351.xml
2023-01-11T21:28:27.6631799Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6631921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6632258Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6632396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6632579Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_dl1zm9z
2023-01-11T21:28:27.6632773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_dl1zm9z/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6633001Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6633006Z 
2023-01-11T21:28:27.6633065Z Running tests...
2023-01-11T21:28:27.6633260Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6633513Z   test_global_local_unused_params_grad (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8248
2023-01-11T21:28:27.6633665Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8249
2023-01-11T21:28:27.6633944Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6634070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6634350Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6634484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6634653Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_fhgbw4s
2023-01-11T21:28:27.6634847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_fhgbw4s/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6635008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6635280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6635406Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6635686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6635817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6635996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp63uijy21
2023-01-11T21:28:27.6636185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp63uijy21/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6636335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6636438Z skip: Need at least 2 CUDA devices (1.759s)
2023-01-11T21:28:27.6636443Z 
2023-01-11T21:28:27.6636635Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6636707Z Ran 1 test in 1.759s
2023-01-11T21:28:27.6636713Z 
2023-01-11T21:28:27.6636781Z OK (skipped=1)
2023-01-11T21:28:27.6636814Z 
2023-01-11T21:28:27.6636900Z Generating XML reports...
2023-01-11T21:28:27.6637244Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212354.xml
2023-01-11T21:28:27.6637511Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6637631Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6637897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6638027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6638210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo56sxyjy
2023-01-11T21:28:27.6638401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo56sxyjy/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6638662Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6638670Z 
2023-01-11T21:28:27.6638744Z Running tests...
2023-01-11T21:28:27.6638937Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6639204Z   test_global_local_unused_params_grad_with_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8281
2023-01-11T21:28:27.6639358Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8282
2023-01-11T21:28:27.6639616Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6639735Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6640012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6640144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6640331Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw7bwecwa
2023-01-11T21:28:27.6640523Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw7bwecwa/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6640791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6640913Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6641176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6641309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6641490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm0kzgm_o
2023-01-11T21:28:27.6641682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm0kzgm_o/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6641846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6642004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6642106Z skip: Need at least 2 CUDA devices (1.765s)
2023-01-11T21:28:27.6642111Z 
2023-01-11T21:28:27.6642303Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6642376Z Ran 1 test in 1.765s
2023-01-11T21:28:27.6642381Z 
2023-01-11T21:28:27.6642440Z OK (skipped=1)
2023-01-11T21:28:27.6642444Z 
2023-01-11T21:28:27.6642527Z Generating XML reports...
2023-01-11T21:28:27.6642872Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212357.xml
2023-01-11T21:28:27.6643140Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6643260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6643590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6643724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6643904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9l78okt7
2023-01-11T21:28:27.6644081Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9l78okt7/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6644308Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6644313Z 
2023-01-11T21:28:27.6644382Z Running tests...
2023-01-11T21:28:27.6644578Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6644844Z   test_global_local_unused_params_grad_with_static_graph (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8314
2023-01-11T21:28:27.6645027Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8315
2023-01-11T21:28:27.6645296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6645417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6645692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6645813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6645998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7fs1g0vk
2023-01-11T21:28:27.6646188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7fs1g0vk/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6646456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6646578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6646854Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6646986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6647167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpht6re9my
2023-01-11T21:28:27.6647357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpht6re9my/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6647506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6647666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6647768Z skip: Need at least 2 CUDA devices (1.756s)
2023-01-11T21:28:27.6647774Z 
2023-01-11T21:28:27.6647967Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6648039Z Ran 1 test in 1.757s
2023-01-11T21:28:27.6648048Z 
2023-01-11T21:28:27.6648119Z OK (skipped=1)
2023-01-11T21:28:27.6648124Z 
2023-01-11T21:28:27.6648203Z Generating XML reports...
2023-01-11T21:28:27.6648548Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212401.xml
2023-01-11T21:28:27.6648807Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6648926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6649203Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6649338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6649518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2phzz64u
2023-01-11T21:28:27.6649712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2phzz64u/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6649971Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6649976Z 
2023-01-11T21:28:27.6650045Z Running tests...
2023-01-11T21:28:27.6650242Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6650493Z   test_gloo_backend_1gpu_module_device_ids_integer_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8347
2023-01-11T21:28:27.6650648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8348
2023-01-11T21:28:27.6651097Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6651223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6651562Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6651697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6651882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk799958h
2023-01-11T21:28:27.6652073Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk799958h/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6652344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6652452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6652728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6652858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6653043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphf3nul4e
2023-01-11T21:28:27.6653240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphf3nul4e/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6653402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6653561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6653663Z skip: Need at least 2 CUDA devices (1.859s)
2023-01-11T21:28:27.6653668Z 
2023-01-11T21:28:27.6653850Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6653922Z Ran 1 test in 1.859s
2023-01-11T21:28:27.6653928Z 
2023-01-11T21:28:27.6653997Z OK (skipped=1)
2023-01-11T21:28:27.6654002Z 
2023-01-11T21:28:27.6654083Z Generating XML reports...
2023-01-11T21:28:27.6654428Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212404.xml
2023-01-11T21:28:27.6654699Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6654826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6655105Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6655237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6655407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3fufgv3q
2023-01-11T21:28:27.6655600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3fufgv3q/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6655823Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6655829Z 
2023-01-11T21:28:27.6655902Z Running tests...
2023-01-11T21:28:27.6656093Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6656364Z   test_gloo_backend_1gpu_module_device_ids_torch_device_list (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8380
2023-01-11T21:28:27.6656553Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8381
2023-01-11T21:28:27.6656821Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6656940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6657209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6657338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6657518Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxklfssgl
2023-01-11T21:28:27.6657710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxklfssgl/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6657909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6658182Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6658301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6658578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6658700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6658882Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpifshma80
2023-01-11T21:28:27.6659076Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpifshma80/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6659234Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6659334Z skip: Need at least 2 CUDA devices (1.761s)
2023-01-11T21:28:27.6659340Z 
2023-01-11T21:28:27.6659537Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6659610Z Ran 1 test in 1.762s
2023-01-11T21:28:27.6659615Z 
2023-01-11T21:28:27.6659685Z OK (skipped=1)
2023-01-11T21:28:27.6659690Z 
2023-01-11T21:28:27.6659772Z Generating XML reports...
2023-01-11T21:28:27.6660105Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212407.xml
2023-01-11T21:28:27.6660378Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6660500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6660779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6660913Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6661098Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjg26_qpp
2023-01-11T21:28:27.6661293Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjg26_qpp/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6661519Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6661524Z 
2023-01-11T21:28:27.6661595Z Running tests...
2023-01-11T21:28:27.6661774Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6662017Z   test_gloo_backend_2gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8413
2023-01-11T21:28:27.6662169Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8414
2023-01-11T21:28:27.6662438Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6662560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6662892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6663064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6663246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpddh65vrv
2023-01-11T21:28:27.6663427Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpddh65vrv/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6663698Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6663819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6664096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6664228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6664440Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnoetbmyg
2023-01-11T21:28:27.6664638Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnoetbmyg/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6664800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6664957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6665047Z skip: Need at least 4 CUDA devices (1.759s)
2023-01-11T21:28:27.6665053Z 
2023-01-11T21:28:27.6665245Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6665319Z Ran 1 test in 1.759s
2023-01-11T21:28:27.6665326Z 
2023-01-11T21:28:27.6665394Z OK (skipped=1)
2023-01-11T21:28:27.6665398Z 
2023-01-11T21:28:27.6665479Z Generating XML reports...
2023-01-11T21:28:27.6665824Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212411.xml
2023-01-11T21:28:27.6666095Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6666220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6666483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6666615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6666798Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb3uonn82
2023-01-11T21:28:27.6666990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb3uonn82/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6667217Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6667222Z 
2023-01-11T21:28:27.6667291Z Running tests...
2023-01-11T21:28:27.6667483Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6667728Z   test_gloo_backend_4gpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8446
2023-01-11T21:28:27.6667882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8447
2023-01-11T21:28:27.6668139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6668260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6668537Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6668670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6668853Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqaejjk7u
2023-01-11T21:28:27.6669047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqaejjk7u/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6669239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6669507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6669628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6669893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6670024Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6670207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprim55ayp
2023-01-11T21:28:27.6670399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprim55ayp/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6670560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6670660Z skip: Need at least 8 CUDA devices (1.759s)
2023-01-11T21:28:27.6670701Z 
2023-01-11T21:28:27.6670897Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6670972Z Ran 1 test in 1.760s
2023-01-11T21:28:27.6670977Z 
2023-01-11T21:28:27.6671034Z OK (skipped=1)
2023-01-11T21:28:27.6671039Z 
2023-01-11T21:28:27.6671120Z Generating XML reports...
2023-01-11T21:28:27.6671463Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212414.xml
2023-01-11T21:28:27.6671737Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6671856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6672133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6672266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6672451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3t4vieo
2023-01-11T21:28:27.6672644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3t4vieo/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6672860Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6672865Z 
2023-01-11T21:28:27.6672935Z Running tests...
2023-01-11T21:28:27.6673128Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6673365Z   test_gloo_backend_cpu_module (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8479
2023-01-11T21:28:27.6673516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8480
2023-01-11T21:28:27.6673783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6673904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6674184Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6674306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6674487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8ccbb14
2023-01-11T21:28:27.6674678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8ccbb14/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6674836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6675104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6675223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6675497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6675660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6675841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyptgk6uh
2023-01-11T21:28:27.6676024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyptgk6uh/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6676184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6676355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6676530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6676831Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6677127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6677324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:28:27.6677490Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:28:27.6677556Z ok (1.865s)
2023-01-11T21:28:27.6677561Z 
2023-01-11T21:28:27.6677743Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6677816Z Ran 1 test in 1.865s
2023-01-11T21:28:27.6677821Z 
2023-01-11T21:28:27.6677879Z OK
2023-01-11T21:28:27.6677884Z 
2023-01-11T21:28:27.6677967Z Generating XML reports...
2023-01-11T21:28:27.6678312Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212417.xml
2023-01-11T21:28:27.6678582Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6678701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6678982Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6679104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6679284Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_165c29
2023-01-11T21:28:27.6679471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_165c29/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6679696Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6679701Z 
2023-01-11T21:28:27.6679774Z Running tests...
2023-01-11T21:28:27.6679967Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6680222Z   test_gloo_backend_cpu_module_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8520
2023-01-11T21:28:27.6680380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8521
2023-01-11T21:28:27.6680652Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6680761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6681037Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6681171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6681356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq0sus5ux
2023-01-11T21:28:27.6681551Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq0sus5ux/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6681713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6681984Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6682138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6682417Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6682538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6682722Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm1v_6ag5
2023-01-11T21:28:27.6682914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm1v_6ag5/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6683077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6683254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6683428Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6683755Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6684054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6684211Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:28:27.6684378Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:28:27.6684445Z ok (1.861s)
2023-01-11T21:28:27.6684450Z 
2023-01-11T21:28:27.6684646Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6684721Z Ran 1 test in 1.861s
2023-01-11T21:28:27.6684725Z 
2023-01-11T21:28:27.6684785Z OK
2023-01-11T21:28:27.6684789Z 
2023-01-11T21:28:27.6684873Z Generating XML reports...
2023-01-11T21:28:27.6685218Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212421.xml
2023-01-11T21:28:27.6685492Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6685602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6685880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6686014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6686200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9wdk2fd6
2023-01-11T21:28:27.6686396Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9wdk2fd6/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6686627Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6686632Z 
2023-01-11T21:28:27.6686705Z Running tests...
2023-01-11T21:28:27.6686903Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6687032Z   test_ignored_output (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6687254Z Test that the output of a model can be ignored and that there is no ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8561
2023-01-11T21:28:27.6687407Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8562
2023-01-11T21:28:27.6687679Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6687800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6688078Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6688213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6688396Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp13trdl1i
2023-01-11T21:28:27.6688634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp13trdl1i/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6688785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6689058Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6689181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6689459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6689591Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6689773Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpatuk6xzy
2023-01-11T21:28:27.6689966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpatuk6xzy/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6690154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6690318Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6690494Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6690789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6691318Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6691484Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:28:27.6691652Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:28:27.6691719Z ok (1.861s)
2023-01-11T21:28:27.6691723Z 
2023-01-11T21:28:27.6691920Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6691998Z Ran 1 test in 1.861s
2023-01-11T21:28:27.6692003Z 
2023-01-11T21:28:27.6692052Z OK
2023-01-11T21:28:27.6692057Z 
2023-01-11T21:28:27.6692142Z Generating XML reports...
2023-01-11T21:28:27.6692488Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212424.xml
2023-01-11T21:28:27.6692760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6692882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6693162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6693299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6693486Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkr2bj64x
2023-01-11T21:28:27.6693682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkr2bj64x/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6693903Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6693908Z 
2023-01-11T21:28:27.6693977Z Running tests...
2023-01-11T21:28:27.6694171Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6694339Z   test_ignored_output_with_unused_parameters (__main__.DistributedDataParallelTest)
2023-01-11T21:28:27.6694560Z Test that the output of a model can be ignored and that there is no ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8606
2023-01-11T21:28:27.6694711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8607
2023-01-11T21:28:27.6694976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6695097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6695420Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6695551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6695736Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ymz9dt_
2023-01-11T21:28:27.6695931Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ymz9dt_/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6696093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6696360Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6696479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6696760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6696931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6697102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_pmk1wyr
2023-01-11T21:28:27.6697297Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_pmk1wyr/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6697460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6697633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6697807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6698103Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6698398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6698470Z ok (1.861s)
2023-01-11T21:28:27.6698475Z 
2023-01-11T21:28:27.6698671Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6698733Z Ran 1 test in 1.861s
2023-01-11T21:28:27.6698737Z 
2023-01-11T21:28:27.6698797Z OK
2023-01-11T21:28:27.6698801Z 
2023-01-11T21:28:27.6698883Z Generating XML reports...
2023-01-11T21:28:27.6699226Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212428.xml
2023-01-11T21:28:27.6699497Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6699617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6699897Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6700030Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6700206Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpntbho0i5
2023-01-11T21:28:27.6700401Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpntbho0i5/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6700626Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6700631Z 
2023-01-11T21:28:27.6700702Z Running tests...
2023-01-11T21:28:27.6700898Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6701137Z   test_ignored_sharded_tensor (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8651
2023-01-11T21:28:27.6701290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8652
2023-01-11T21:28:27.6701558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6701679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6701978Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6702112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6702296Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfpmo22hq
2023-01-11T21:28:27.6702490Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfpmo22hq/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6702652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6702994Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6703119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6703402Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6703571Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6703743Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw8820_uo
2023-01-11T21:28:27.6703932Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw8820_uo/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6704093Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6704197Z skip: Need at least 2 CUDA devices (1.864s)
2023-01-11T21:28:27.6704203Z 
2023-01-11T21:28:27.6704402Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6704480Z Ran 1 test in 1.865s
2023-01-11T21:28:27.6704486Z 
2023-01-11T21:28:27.6704556Z OK (skipped=1)
2023-01-11T21:28:27.6704560Z 
2023-01-11T21:28:27.6704644Z Generating XML reports...
2023-01-11T21:28:27.6704977Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212431.xml
2023-01-11T21:28:27.6705252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6705372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6705649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6705784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6705966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf1vvoxpt
2023-01-11T21:28:27.6706160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf1vvoxpt/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6706387Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6706392Z 
2023-01-11T21:28:27.6706461Z Running tests...
2023-01-11T21:28:27.6706649Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6706895Z   test_invalid_powerSGD_state (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8684
2023-01-11T21:28:27.6707047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8685
2023-01-11T21:28:27.6707315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6707435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6707715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6707846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6708029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7zxgfv4
2023-01-11T21:28:27.6708210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7zxgfv4/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6708405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6708816Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:27.6709223Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:27.6709650Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:27.6710056Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:27.6710458Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:27.6710861Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:27.6711138Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6711261Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6711543Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6711677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6711860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwvzpktx1
2023-01-11T21:28:27.6712059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwvzpktx1/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6712222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6712625Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:27.6713024Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:27.6713425Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:27.6713850Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:27.6714249Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:27.6714672Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:28:27.6714731Z ok (1.861s)
2023-01-11T21:28:27.6714748Z 
2023-01-11T21:28:27.6714933Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6715006Z Ran 1 test in 1.862s
2023-01-11T21:28:27.6715011Z 
2023-01-11T21:28:27.6715070Z OK
2023-01-11T21:28:27.6715075Z 
2023-01-11T21:28:27.6715158Z Generating XML reports...
2023-01-11T21:28:27.6715504Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212435.xml
2023-01-11T21:28:27.6715780Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6715908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6716186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6716306Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6716490Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0jpssi75
2023-01-11T21:28:27.6716683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0jpssi75/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6716909Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6716915Z 
2023-01-11T21:28:27.6716987Z Running tests...
2023-01-11T21:28:27.6717183Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6717423Z   test_save_load_checkpoint (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8717
2023-01-11T21:28:27.6717581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8718
2023-01-11T21:28:27.6717850Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6717960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6718238Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6718370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6718553Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_8ihg7ij
2023-01-11T21:28:27.6718745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_8ihg7ij/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6718907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6719180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6719337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6719605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6719738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6719922Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6l7nxn6u
2023-01-11T21:28:27.6720113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6l7nxn6u/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6720273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6720374Z skip: Need at least 2 CUDA devices (1.867s)
2023-01-11T21:28:27.6720379Z 
2023-01-11T21:28:27.6720573Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6720678Z Ran 1 test in 1.867s
2023-01-11T21:28:27.6720684Z 
2023-01-11T21:28:27.6720754Z OK (skipped=1)
2023-01-11T21:28:27.6720759Z 
2023-01-11T21:28:27.6720829Z Generating XML reports...
2023-01-11T21:28:27.6721176Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212438.xml
2023-01-11T21:28:27.6721453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6721575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6721855Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6721988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6722171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn7xii74s
2023-01-11T21:28:27.6722365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn7xii74s/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6722581Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6722600Z 
2023-01-11T21:28:27.6722659Z Running tests...
2023-01-11T21:28:27.6722852Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6723085Z   test_sparse_gradients (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8750
2023-01-11T21:28:27.6723237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8751
2023-01-11T21:28:27.6723505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6723625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6723905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6724041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6724210Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnbv5ka9t
2023-01-11T21:28:27.6724405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnbv5ka9t/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6724566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6724834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6724958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6725235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6725367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6725555Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd1xsrgyj
2023-01-11T21:28:27.6725770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd1xsrgyj/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6725933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6726109Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6726283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6726579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6726877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6726944Z ok (1.866s)
2023-01-11T21:28:27.6726950Z 
2023-01-11T21:28:27.6727174Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6727252Z Ran 1 test in 1.866s
2023-01-11T21:28:27.6727257Z 
2023-01-11T21:28:27.6727304Z OK
2023-01-11T21:28:27.6727321Z 
2023-01-11T21:28:27.6727391Z Generating XML reports...
2023-01-11T21:28:27.6727737Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212442.xml
2023-01-11T21:28:27.6728008Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6728129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6728408Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6728540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6728723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt9bxwp6u
2023-01-11T21:28:27.6728921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt9bxwp6u/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6729136Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6729141Z 
2023-01-11T21:28:27.6729211Z Running tests...
2023-01-11T21:28:27.6729405Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6729654Z   test_sparse_gradients_grad_is_view (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8795
2023-01-11T21:28:27.6729807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8796
2023-01-11T21:28:27.6730075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6730197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6730476Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6730613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6730787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcwb75qhu
2023-01-11T21:28:27.6731250Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcwb75qhu/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6731526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6731649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6731927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6732062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6732244Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy8qsjd9z
2023-01-11T21:28:27.6732544Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy8qsjd9z/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6732695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6732859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6733036Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6733211Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6733512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6733808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:28:27.6733876Z ok (1.866s)
2023-01-11T21:28:27.6733882Z 
2023-01-11T21:28:27.6734117Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6734195Z Ran 1 test in 1.866s
2023-01-11T21:28:27.6734200Z 
2023-01-11T21:28:27.6734247Z OK
2023-01-11T21:28:27.6734252Z 
2023-01-11T21:28:27.6734336Z Generating XML reports...
2023-01-11T21:28:27.6734678Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212445.xml
2023-01-11T21:28:27.6734949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6735070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6735347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6735483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6735667Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv2hfar3j
2023-01-11T21:28:27.6735853Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv2hfar3j/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6736079Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6736085Z 
2023-01-11T21:28:27.6736158Z Running tests...
2023-01-11T21:28:27.6736353Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6736598Z   test_sync_batch_norm_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8840
2023-01-11T21:28:27.6736751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8841
2023-01-11T21:28:27.6737021Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6737143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6737421Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6737543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6737729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyjhxpwx1
2023-01-11T21:28:27.6737926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyjhxpwx1/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6738195Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6738318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6738600Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6738733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6738918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxefx206s
2023-01-11T21:28:27.6739146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxefx206s/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6739297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6739460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6739564Z skip: Need at least 2 CUDA devices (1.863s)
2023-01-11T21:28:27.6739569Z 
2023-01-11T21:28:27.6739767Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6739845Z Ran 1 test in 1.863s
2023-01-11T21:28:27.6739850Z 
2023-01-11T21:28:27.6739921Z OK (skipped=1)
2023-01-11T21:28:27.6739925Z 
2023-01-11T21:28:27.6740010Z Generating XML reports...
2023-01-11T21:28:27.6740356Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212448.xml
2023-01-11T21:28:27.6740651Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6740777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6741056Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6741189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6741374Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyy2l8asf
2023-01-11T21:28:27.6741570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyy2l8asf/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6741800Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6741806Z 
2023-01-11T21:28:27.6741878Z Running tests...
2023-01-11T21:28:27.6742076Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6742312Z   test_sync_batch_norm_only_empty_input (__main__.DistributedDataParallelTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8873
2023-01-11T21:28:27.6742470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8874
2023-01-11T21:28:27.6742745Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6742945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6743229Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6743362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6743547Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyygbsutu
2023-01-11T21:28:27.6743745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyygbsutu/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6744020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6744130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6744410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6744543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6744728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmlfbas8v
2023-01-11T21:28:27.6744920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmlfbas8v/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6745083Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6745245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6745349Z skip: Need at least 2 CUDA devices (1.859s)
2023-01-11T21:28:27.6745393Z 
2023-01-11T21:28:27.6745576Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6745657Z Ran 1 test in 1.859s
2023-01-11T21:28:27.6745662Z 
2023-01-11T21:28:27.6745732Z OK (skipped=1)
2023-01-11T21:28:27.6745737Z 
2023-01-11T21:28:27.6745820Z Generating XML reports...
2023-01-11T21:28:27.6746163Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212452.xml
2023-01-11T21:28:27.6746432Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6746553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6746833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6746965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6747167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphyndggio
2023-01-11T21:28:27.6747363Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphyndggio/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6747591Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6747597Z 
2023-01-11T21:28:27.6747667Z Running tests...
2023-01-11T21:28:27.6747860Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6748139Z   test_all_to_all_single (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8906
2023-01-11T21:28:27.6748410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6748531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6748813Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6748938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6749122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxry0mima
2023-01-11T21:28:27.6749314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxry0mima/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6749477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6749655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6749956Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:28:27.6750022Z ok (1.759s)
2023-01-11T21:28:27.6750027Z 
2023-01-11T21:28:27.6750219Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6750281Z Ran 1 test in 1.760s
2023-01-11T21:28:27.6750291Z 
2023-01-11T21:28:27.6750349Z OK
2023-01-11T21:28:27.6750354Z 
2023-01-11T21:28:27.6750437Z Generating XML reports...
2023-01-11T21:28:27.6750858Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20230111212455.xml
2023-01-11T21:28:27.6751126Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6751246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6751524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6751656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6751839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj57m6sda
2023-01-11T21:28:27.6752024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj57m6sda/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6752288Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6752293Z 
2023-01-11T21:28:27.6752363Z Running tests...
2023-01-11T21:28:27.6752558Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6752845Z   test_allgather_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8931
2023-01-11T21:28:27.6753118Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6753238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6753514Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6753645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6753847Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp73bk0poe
2023-01-11T21:28:27.6754042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp73bk0poe/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6754202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6754376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6754673Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:28:27.6755257Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:28:27.6755410Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:28:27.6755478Z ok (1.754s)
2023-01-11T21:28:27.6755483Z 
2023-01-11T21:28:27.6755675Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6755738Z Ran 1 test in 1.755s
2023-01-11T21:28:27.6755743Z 
2023-01-11T21:28:27.6755802Z OK
2023-01-11T21:28:27.6755807Z 
2023-01-11T21:28:27.6755884Z Generating XML reports...
2023-01-11T21:28:27.6768782Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20230111212459.xml
2023-01-11T21:28:27.6769173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6769300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6769590Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6769737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6769924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplvewvwta
2023-01-11T21:28:27.6770120Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplvewvwta/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6770341Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6770365Z 
2023-01-11T21:28:27.6770425Z Running tests...
2023-01-11T21:28:27.6770625Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6771146Z   test_allreduce_coalesced (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8956
2023-01-11T21:28:27.6771473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6771601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6772053Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6772188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6772375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvefkiis
2023-01-11T21:28:27.6772559Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvefkiis/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6772724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6772899Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6773200Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:28:27.6773813Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:28:27.6773971Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:28:27.6774038Z ok (1.755s)
2023-01-11T21:28:27.6774043Z 
2023-01-11T21:28:27.6774241Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6774315Z Ran 1 test in 1.755s
2023-01-11T21:28:27.6774321Z 
2023-01-11T21:28:27.6774368Z OK
2023-01-11T21:28:27.6774384Z 
2023-01-11T21:28:27.6774455Z Generating XML reports...
2023-01-11T21:28:27.6774877Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20230111212502.xml
2023-01-11T21:28:27.6775152Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6775281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6775559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6775694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6775877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp63sexbqg
2023-01-11T21:28:27.6776072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp63sexbqg/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6776288Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6776309Z 
2023-01-11T21:28:27.6776370Z Running tests...
2023-01-11T21:28:27.6776564Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6776847Z   test_collectives (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8981
2023-01-11T21:28:27.6777123Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6777244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6777524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6777656Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6777841Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc332pdmz
2023-01-11T21:28:27.6778022Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc332pdmz/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6778185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6778359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6778694Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:28:27.6778761Z ok (1.759s)
2023-01-11T21:28:27.6778766Z 
2023-01-11T21:28:27.6778958Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6779032Z Ran 1 test in 1.759s
2023-01-11T21:28:27.6779037Z 
2023-01-11T21:28:27.6779096Z OK
2023-01-11T21:28:27.6779101Z 
2023-01-11T21:28:27.6779170Z Generating XML reports...
2023-01-11T21:28:27.6779591Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20230111212505.xml
2023-01-11T21:28:27.6779862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6779984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6780296Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6780433Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6780615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbl9hoh3q
2023-01-11T21:28:27.6780809Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbl9hoh3q/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6781041Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6781047Z 
2023-01-11T21:28:27.6781105Z Running tests...
2023-01-11T21:28:27.6781301Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6781589Z   test_monitored_barrier (__main__.GlooProcessGroupWithDispatchedCollectivesTests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9006
2023-01-11T21:28:27.6781862Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6781988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6782270Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6782403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6782584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuhmuci9o
2023-01-11T21:28:27.6782778Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuhmuci9o/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6783008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6783187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6783488Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:28:27.6783560Z ok (1.757s)
2023-01-11T21:28:27.6783566Z 
2023-01-11T21:28:27.6783763Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6783838Z Ran 1 test in 1.757s
2023-01-11T21:28:27.6783843Z 
2023-01-11T21:28:27.6783904Z OK
2023-01-11T21:28:27.6783908Z 
2023-01-11T21:28:27.6783991Z Generating XML reports...
2023-01-11T21:28:27.6784396Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20230111212509.xml
2023-01-11T21:28:27.6784671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6784795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6785075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6785208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6785433Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsrgpendm
2023-01-11T21:28:27.6785627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsrgpendm/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6785855Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6785861Z 
2023-01-11T21:28:27.6785932Z Running tests...
2023-01-11T21:28:27.6786114Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6786340Z   test_allgather_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9031
2023-01-11T21:28:27.6786493Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9032
2023-01-11T21:28:27.6786647Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9033
2023-01-11T21:28:27.6786825Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9034
2023-01-11T21:28:27.6787104Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6787224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6787507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6787639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6787809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4rlk_ga4
2023-01-11T21:28:27.6788001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4rlk_ga4/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6788271Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6788392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6788674Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6788807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6788989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptazb_3__
2023-01-11T21:28:27.6789185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptazb_3__/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6789441Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6789563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6789840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6789971Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6790155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4eus9fdf
2023-01-11T21:28:27.6790349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4eus9fdf/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6790617Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6790739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6791015Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6791136Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6791321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu2fidlyg
2023-01-11T21:28:27.6791512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu2fidlyg/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6791674Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6791884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6792044Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6792204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6792271Z ok (2.067s)
2023-01-11T21:28:27.6792277Z 
2023-01-11T21:28:27.6792461Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6792536Z Ran 1 test in 2.067s
2023-01-11T21:28:27.6792542Z 
2023-01-11T21:28:27.6792599Z OK
2023-01-11T21:28:27.6792604Z 
2023-01-11T21:28:27.6792687Z Generating XML reports...
2023-01-11T21:28:27.6793011Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212512.xml
2023-01-11T21:28:27.6793310Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6793436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6793715Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6793847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6794017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3jo7egy_
2023-01-11T21:28:27.6794210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3jo7egy_/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6794439Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6794445Z 
2023-01-11T21:28:27.6794518Z Running tests...
2023-01-11T21:28:27.6794717Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6794948Z   test_allgather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9098
2023-01-11T21:28:27.6795106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9099
2023-01-11T21:28:27.6795257Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9100
2023-01-11T21:28:27.6795394Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9101
2023-01-11T21:28:27.6795663Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6795785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6796066Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6796199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6796384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyz9elro3
2023-01-11T21:28:27.6796580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyz9elro3/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6796740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6797009Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6797118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6797394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6797526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6797707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp11s7kt0r
2023-01-11T21:28:27.6797897Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp11s7kt0r/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6798059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6798363Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6798485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6798749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6798883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6799065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ud3pwcq
2023-01-11T21:28:27.6799256Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ud3pwcq/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6799519Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6799641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6799953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6800088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6800272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp76oaa4cw
2023-01-11T21:28:27.6800420Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6800615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp76oaa4cw/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6800775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6800877Z skip: Need at least 2 CUDA devices (1.964s)
2023-01-11T21:28:27.6800882Z 
2023-01-11T21:28:27.6801080Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6801154Z Ran 1 test in 1.965s
2023-01-11T21:28:27.6801162Z 
2023-01-11T21:28:27.6801235Z OK (skipped=1)
2023-01-11T21:28:27.6801240Z 
2023-01-11T21:28:27.6801323Z Generating XML reports...
2023-01-11T21:28:27.6801634Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212515.xml
2023-01-11T21:28:27.6801909Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6802030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6802305Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6802437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6802621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8x_zphf9
2023-01-11T21:28:27.6802813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8x_zphf9/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6803047Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6803053Z 
2023-01-11T21:28:27.6803127Z Running tests...
2023-01-11T21:28:27.6803308Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6803532Z   test_allgather_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9153
2023-01-11T21:28:27.6803687Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9154
2023-01-11T21:28:27.6803837Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9155
2023-01-11T21:28:27.6803986Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9156
2023-01-11T21:28:27.6804259Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6804382Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6804700Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6804832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6805003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4dq3ryes
2023-01-11T21:28:27.6805198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4dq3ryes/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6805358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6805630Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6805750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6806029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6806194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6806380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvbllqaj1
2023-01-11T21:28:27.6806561Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvbllqaj1/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6806724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6806995Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6807115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6807392Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6807524Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6807707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5u1as2t3
2023-01-11T21:28:27.6807898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5u1as2t3/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6808057Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6808314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6808435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6808714Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6808846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6809027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw3cb6sy0
2023-01-11T21:28:27.6809219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw3cb6sy0/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6809384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6809450Z ok (1.970s)
2023-01-11T21:28:27.6809456Z 
2023-01-11T21:28:27.6809639Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6809714Z Ran 1 test in 1.970s
2023-01-11T21:28:27.6809719Z 
2023-01-11T21:28:27.6809778Z OK
2023-01-11T21:28:27.6809783Z 
2023-01-11T21:28:27.6809865Z Generating XML reports...
2023-01-11T21:28:27.6810186Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212519.xml
2023-01-11T21:28:27.6810456Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6810577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6810990Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6811234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6811407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpedd6sogl
2023-01-11T21:28:27.6811604Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpedd6sogl/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6811842Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6811848Z 
2023-01-11T21:28:27.6811920Z Running tests...
2023-01-11T21:28:27.6812115Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6812350Z   test_allgather_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9220
2023-01-11T21:28:27.6812504Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9221
2023-01-11T21:28:27.6812693Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9222
2023-01-11T21:28:27.6812835Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9223
2023-01-11T21:28:27.6813108Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6813229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6813508Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6813642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6813827Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiryv96bz
2023-01-11T21:28:27.6814021Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiryv96bz/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6814181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6814453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6814564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6814843Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6814975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6815159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzvg9bsar
2023-01-11T21:28:27.6815352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzvg9bsar/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6815512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6815784Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6815907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6816173Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6816304Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6816492Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpznwy3qsi
2023-01-11T21:28:27.6816684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpznwy3qsi/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6816953Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6817073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6817348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6817479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6817693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpolb5fd1b
2023-01-11T21:28:27.6817872Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpolb5fd1b/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6818033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6818192Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6818367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6818541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6818712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:28:27.6818882Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:28:27.6819214Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:28:27.6819515Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:28:27.6819798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:28:27.6820084Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:28:27.6820657Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:28:27.6820807Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:28:27.6821366Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:28:27.6821518Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:28:27.6822075Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:28:27.6822219Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:28:27.6822767Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:28:27.6822997Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:28:27.6823067Z ok (2.063s)
2023-01-11T21:28:27.6823073Z 
2023-01-11T21:28:27.6823269Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6823332Z Ran 1 test in 2.063s
2023-01-11T21:28:27.6823338Z 
2023-01-11T21:28:27.6823397Z OK
2023-01-11T21:28:27.6823402Z 
2023-01-11T21:28:27.6823485Z Generating XML reports...
2023-01-11T21:28:27.6823809Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212523.xml
2023-01-11T21:28:27.6824083Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6824206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6824487Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6824655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6824829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmputlv1q28
2023-01-11T21:28:27.6825024Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmputlv1q28/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6825253Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6825258Z 
2023-01-11T21:28:27.6825331Z Running tests...
2023-01-11T21:28:27.6825528Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6825765Z   test_allgather_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9287
2023-01-11T21:28:27.6825946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9288
2023-01-11T21:28:27.6826100Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9289
2023-01-11T21:28:27.6826249Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9290
2023-01-11T21:28:27.6826509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6826632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6826911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6827046Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6827231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptn2pn_kx
2023-01-11T21:28:27.6827425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptn2pn_kx/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6827589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6827865Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6827974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6828252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6828385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6828568Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpot0oq5q2
2023-01-11T21:28:27.6828761Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpot0oq5q2/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6828922Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6829191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6829320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6829587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6829719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6829904Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsr5xx0dy
2023-01-11T21:28:27.6830096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsr5xx0dy/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6830256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6830525Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6830645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6830955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6831076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6831260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptpnfhq7b
2023-01-11T21:28:27.6831457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptpnfhq7b/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6831618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6832178Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:28:27.6832332Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:28:27.6832913Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:28:27.6833062Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:28:27.6833618Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:28:27.6833762Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:28:27.6834309Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:2589: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:28:27.6834457Z   "torch.distributed.all_gather_coalesced will be deprecated. If you must "
2023-01-11T21:28:27.6834510Z ok (2.173s)
2023-01-11T21:28:27.6834516Z 
2023-01-11T21:28:27.6834712Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6834787Z Ran 1 test in 2.174s
2023-01-11T21:28:27.6834792Z 
2023-01-11T21:28:27.6834850Z OK
2023-01-11T21:28:27.6834854Z 
2023-01-11T21:28:27.6834937Z Generating XML reports...
2023-01-11T21:28:27.6835262Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212526.xml
2023-01-11T21:28:27.6835531Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6835654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6835925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6836058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6836245Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq8kywtvp
2023-01-11T21:28:27.6836441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq8kywtvp/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6836670Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6836675Z 
2023-01-11T21:28:27.6836746Z Running tests...
2023-01-11T21:28:27.6836943Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6837180Z   test_allgather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9354
2023-01-11T21:28:27.6837337Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9355
2023-01-11T21:28:27.6837527Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9356
2023-01-11T21:28:27.6837675Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9357
2023-01-11T21:28:27.6837949Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6838070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6838349Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6838482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6838665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpijmzmcqt
2023-01-11T21:28:27.6838863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpijmzmcqt/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6839150Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6839275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6839555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6839687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6839869Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp86xj3thy
2023-01-11T21:28:27.6840060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp86xj3thy/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6840327Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6840448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6840727Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6840849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6841033Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwtcr8y7t
2023-01-11T21:28:27.6841227Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwtcr8y7t/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6841390Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6841555Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6841709Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6841976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6842098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6842380Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6842500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6842682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd1gcze7y
2023-01-11T21:28:27.6842873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd1gcze7y/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6843035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6843101Z ok (2.069s)
2023-01-11T21:28:27.6843107Z 
2023-01-11T21:28:27.6843303Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6843377Z Ran 1 test in 2.069s
2023-01-11T21:28:27.6843382Z 
2023-01-11T21:28:27.6843441Z OK
2023-01-11T21:28:27.6843446Z 
2023-01-11T21:28:27.6843515Z Generating XML reports...
2023-01-11T21:28:27.6843844Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212530.xml
2023-01-11T21:28:27.6844144Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6844266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6844542Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6844675Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6844857Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8a7400vy
2023-01-11T21:28:27.6845049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8a7400vy/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6845265Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6845284Z 
2023-01-11T21:28:27.6845344Z Running tests...
2023-01-11T21:28:27.6845597Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6845821Z   test_allgather_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9421
2023-01-11T21:28:27.6845973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9422
2023-01-11T21:28:27.6846123Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9423
2023-01-11T21:28:27.6846270Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9424
2023-01-11T21:28:27.6846544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6846666Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6846935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6847074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6847259Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6f8405o2
2023-01-11T21:28:27.6847449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6f8405o2/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6847609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6847880Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6848002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6848280Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6848413Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6848584Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6uprkvdz
2023-01-11T21:28:27.6848781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6uprkvdz/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6849046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6849167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6849446Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6849579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6849760Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5537emkq
2023-01-11T21:28:27.6849949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5537emkq/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6850206Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6850356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6850632Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6850763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6851215Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9m6ojrqd
2023-01-11T21:28:27.6851412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9m6ojrqd/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6851576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6851738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6851900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6851953Z ok (3.371s)
2023-01-11T21:28:27.6851961Z 
2023-01-11T21:28:27.6852223Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6852299Z Ran 1 test in 3.372s
2023-01-11T21:28:27.6852304Z 
2023-01-11T21:28:27.6852366Z OK
2023-01-11T21:28:27.6852370Z 
2023-01-11T21:28:27.6852454Z Generating XML reports...
2023-01-11T21:28:27.6852788Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212534.xml
2023-01-11T21:28:27.6853060Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6853183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6853448Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6853579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6853764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5k9unizr
2023-01-11T21:28:27.6853960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5k9unizr/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6854188Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6854193Z 
2023-01-11T21:28:27.6854265Z Running tests...
2023-01-11T21:28:27.6854462Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6854693Z   test_allgather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9512
2023-01-11T21:28:27.6854846Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9513
2023-01-11T21:28:27.6854984Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9514
2023-01-11T21:28:27.6855133Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9515
2023-01-11T21:28:27.6855406Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6855530Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6855812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6855945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6856131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkf338mvp
2023-01-11T21:28:27.6856400Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6856508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6856787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6856924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6857145Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphq6iyluu
2023-01-11T21:28:27.6857340Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkf338mvp/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6857529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphq6iyluu/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6857692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6857848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6858119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6858229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6858505Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6858674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6858860Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9q83defu
2023-01-11T21:28:27.6859056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9q83defu/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6859217Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6859488Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6859611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6859876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6860010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6860195Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9uxa71xq
2023-01-11T21:28:27.6860389Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9uxa71xq/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6860548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6860649Z skip: Need at least 2 CUDA devices (2.070s)
2023-01-11T21:28:27.6860654Z 
2023-01-11T21:28:27.6860850Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6860927Z Ran 1 test in 2.070s
2023-01-11T21:28:27.6860931Z 
2023-01-11T21:28:27.6861006Z OK (skipped=1)
2023-01-11T21:28:27.6861011Z 
2023-01-11T21:28:27.6861082Z Generating XML reports...
2023-01-11T21:28:27.6861406Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212539.xml
2023-01-11T21:28:27.6861677Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6861805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6862085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6862220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6862404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq1j08suu
2023-01-11T21:28:27.6862597Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq1j08suu/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6862880Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6862901Z 
2023-01-11T21:28:27.6862963Z Running tests...
2023-01-11T21:28:27.6863160Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6863386Z   test_allreduce_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9567
2023-01-11T21:28:27.6863581Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9568
2023-01-11T21:28:27.6863733Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9569
2023-01-11T21:28:27.6863886Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9570
2023-01-11T21:28:27.6864158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6864283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6864549Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6864684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6864867Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9lrguymy
2023-01-11T21:28:27.6865093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9lrguymy/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6865369Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6865490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6865767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6865898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6866082Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp94hgmon
2023-01-11T21:28:27.6866262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp94hgmon/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6866425Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6866585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6866861Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6866983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6867264Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6867398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6867582Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptupwyr3k
2023-01-11T21:28:27.6867765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptupwyr3k/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6867929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6868197Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6868322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6868596Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6868729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6868909Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ukxhkms
2023-01-11T21:28:27.6869102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ukxhkms/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6869264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6869318Z ok (2.071s)
2023-01-11T21:28:27.6869324Z 
2023-01-11T21:28:27.6869519Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6869593Z Ran 1 test in 2.071s
2023-01-11T21:28:27.6869599Z 
2023-01-11T21:28:27.6869659Z OK
2023-01-11T21:28:27.6869697Z 
2023-01-11T21:28:27.6869782Z Generating XML reports...
2023-01-11T21:28:27.6870105Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212542.xml
2023-01-11T21:28:27.6870375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6870495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6870760Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6870892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6871075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoz03d7dn
2023-01-11T21:28:27.6871265Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoz03d7dn/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6871526Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6871535Z 
2023-01-11T21:28:27.6871607Z Running tests...
2023-01-11T21:28:27.6871800Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6872030Z   test_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9634
2023-01-11T21:28:27.6872182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9635
2023-01-11T21:28:27.6872320Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9636
2023-01-11T21:28:27.6872470Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9637
2023-01-11T21:28:27.6872741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6872862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6873142Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6873276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6873459Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx629em5c
2023-01-11T21:28:27.6873650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx629em5c/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6873800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6874068Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6874188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6874466Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6874597Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6874789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd4sj1paz
2023-01-11T21:28:27.6874986Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd4sj1paz/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6875147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6875416Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6875526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6875800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6875931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6876111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcgdhehdb
2023-01-11T21:28:27.6876338Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcgdhehdb/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6876605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6876725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6877001Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6877122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6877304Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxiwaxhm8
2023-01-11T21:28:27.6877501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxiwaxhm8/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6877664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6877852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6877957Z skip: Need at least 2 CUDA devices (1.969s)
2023-01-11T21:28:27.6877963Z 
2023-01-11T21:28:27.6878158Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6878232Z Ran 1 test in 1.969s
2023-01-11T21:28:27.6878237Z 
2023-01-11T21:28:27.6878306Z OK (skipped=1)
2023-01-11T21:28:27.6878311Z 
2023-01-11T21:28:27.6878382Z Generating XML reports...
2023-01-11T21:28:27.6878705Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212546.xml
2023-01-11T21:28:27.6878976Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6879099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6879377Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6879513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6879698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1duot8yo
2023-01-11T21:28:27.6879889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1duot8yo/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6880105Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6880122Z 
2023-01-11T21:28:27.6880181Z Running tests...
2023-01-11T21:28:27.6880377Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6880619Z   test_allreduce_basics_cuda_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9689
2023-01-11T21:28:27.6880772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9690
2023-01-11T21:28:27.6880922Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9691
2023-01-11T21:28:27.6881075Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9692
2023-01-11T21:28:27.6881343Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6881465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6881728Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6881860Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6882043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps262taw4
2023-01-11T21:28:27.6882234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps262taw4/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6882393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6882661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6882845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6883125Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6883256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6883428Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmxco6ftw
2023-01-11T21:28:27.6883624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmxco6ftw/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6883788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6884055Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6884176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6884485Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6884619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6884802Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpakg83idq
2023-01-11T21:28:27.6884982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpakg83idq/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6885253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6885375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6885649Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6885781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6885970Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmply1zfe9k
2023-01-11T21:28:27.6886160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmply1zfe9k/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6886323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6886481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6886571Z skip: Need at least 2 CUDA devices (1.870s)
2023-01-11T21:28:27.6886576Z 
2023-01-11T21:28:27.6886770Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6886843Z Ran 1 test in 1.870s
2023-01-11T21:28:27.6886848Z 
2023-01-11T21:28:27.6886919Z OK (skipped=1)
2023-01-11T21:28:27.6886923Z 
2023-01-11T21:28:27.6887008Z Generating XML reports...
2023-01-11T21:28:27.6887332Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212549.xml
2023-01-11T21:28:27.6887602Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6887723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6887987Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6888118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6888303Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptbh4g1ku
2023-01-11T21:28:27.6888495Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptbh4g1ku/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6888725Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6888731Z 
2023-01-11T21:28:27.6888801Z Running tests...
2023-01-11T21:28:27.6888998Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6889268Z   test_allreduce_basics_using_work_api (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9744
2023-01-11T21:28:27.6889419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9745
2023-01-11T21:28:27.6889557Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9746
2023-01-11T21:28:27.6889706Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9747
2023-01-11T21:28:27.6889974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6890096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6890376Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6890506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6890728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsso6gbd1
2023-01-11T21:28:27.6891130Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsso6gbd1/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6891293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6891570Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6891690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6891969Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6892104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6892287Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4zgt8e6l
2023-01-11T21:28:27.6892482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4zgt8e6l/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6892647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6892917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6893027Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6893307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6893439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6893622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlfgmdqi
2023-01-11T21:28:27.6893816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlfgmdqi/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6894085Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6894207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6894483Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6894613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6894783Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeqaiyjd8
2023-01-11T21:28:27.6894975Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeqaiyjd8/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6895136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6895296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6895363Z ok (2.066s)
2023-01-11T21:28:27.6895368Z 
2023-01-11T21:28:27.6895566Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6895694Z Ran 1 test in 2.067s
2023-01-11T21:28:27.6895699Z 
2023-01-11T21:28:27.6895760Z OK
2023-01-11T21:28:27.6895765Z 
2023-01-11T21:28:27.6895835Z Generating XML reports...
2023-01-11T21:28:27.6896163Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212553.xml
2023-01-11T21:28:27.6896434Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6896556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6896833Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6896966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6897148Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpshj2_k7x
2023-01-11T21:28:27.6897377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpshj2_k7x/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6897607Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6897613Z 
2023-01-11T21:28:27.6897672Z Running tests...
2023-01-11T21:28:27.6897867Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6898092Z   test_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9811
2023-01-11T21:28:27.6898246Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9812
2023-01-11T21:28:27.6898400Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9813
2023-01-11T21:28:27.6898552Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9814
2023-01-11T21:28:27.6898824Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6898949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6899216Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6899348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6899534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpurptpd25
2023-01-11T21:28:27.6899728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpurptpd25/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6899891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6900158Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6900278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6900557Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6900689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6900858Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp703j6u58
2023-01-11T21:28:27.6901049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp703j6u58/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6901212Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6901480Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6901601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6901878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6902012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6902228Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmycs01q5
2023-01-11T21:28:27.6902410Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmycs01q5/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6902573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6902905Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6903031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6903314Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6903447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6903629Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3lex4qef
2023-01-11T21:28:27.6903854Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3lex4qef/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6904016Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6904070Z ok (2.068s)
2023-01-11T21:28:27.6904076Z 
2023-01-11T21:28:27.6904277Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6904353Z Ran 1 test in 2.068s
2023-01-11T21:28:27.6904358Z 
2023-01-11T21:28:27.6904420Z OK
2023-01-11T21:28:27.6904425Z 
2023-01-11T21:28:27.6904511Z Generating XML reports...
2023-01-11T21:28:27.6904834Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212557.xml
2023-01-11T21:28:27.6905109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6905234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6905500Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6905634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6905816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdihuoxz1
2023-01-11T21:28:27.6906009Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdihuoxz1/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6906236Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6906241Z 
2023-01-11T21:28:27.6906312Z Running tests...
2023-01-11T21:28:27.6906504Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6906736Z   test_allreduce_coalesced_async (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9878
2023-01-11T21:28:27.6906887Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9879
2023-01-11T21:28:27.6907029Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9880
2023-01-11T21:28:27.6907177Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9881
2023-01-11T21:28:27.6907447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6907567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6907845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6907976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6908159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpakurfxcc
2023-01-11T21:28:27.6908352Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpakurfxcc/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6908504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6908808Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6908928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6909208Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6909339Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6909519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpasobml84
2023-01-11T21:28:27.6909712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpasobml84/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6909875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6910176Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6910289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6910569Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6910706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6910892Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa4lnlwa2
2023-01-11T21:28:27.6911085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa4lnlwa2/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6911357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6911482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6911761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6911885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6912070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprekc_ncr
2023-01-11T21:28:27.6912264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprekc_ncr/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6912429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6912589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6912768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.6912943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:28:27.6913119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:28:27.6913296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.6913583Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:28:27.6913884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:28:27.6914172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:28:27.6914469Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:28:27.6915034Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:28:27.6915217Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:28:27.6915766Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:28:27.6915914Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:28:27.6916465Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:28:27.6916609Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:28:27.6917188Z /opt/conda/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py:1715: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:28:27.6917340Z   "torch.distributed.all_reduce_coalesced will be deprecated. If you must "
2023-01-11T21:28:27.6917395Z ok (1.967s)
2023-01-11T21:28:27.6917400Z 
2023-01-11T21:28:27.6917597Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6917672Z Ran 1 test in 1.967s
2023-01-11T21:28:27.6917677Z 
2023-01-11T21:28:27.6917738Z OK
2023-01-11T21:28:27.6917743Z 
2023-01-11T21:28:27.6917827Z Generating XML reports...
2023-01-11T21:28:27.6918147Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212600.xml
2023-01-11T21:28:27.6918418Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6918546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6918825Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6918947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6919134Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2dfb1z17
2023-01-11T21:28:27.6919329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2dfb1z17/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6919563Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6919568Z 
2023-01-11T21:28:27.6919641Z Running tests...
2023-01-11T21:28:27.6919838Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6920073Z   test_allreduce_coalesced_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9945
2023-01-11T21:28:27.6920232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9946
2023-01-11T21:28:27.6920372Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 9947
2023-01-11T21:28:27.6920524Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 9948
2023-01-11T21:28:27.6920797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6920917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6921199Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6921335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6921520Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp73ff2mb6
2023-01-11T21:28:27.6921713Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp73ff2mb6/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6922016Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6922124Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6922397Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6922529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6922712Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4vhf7mo2
2023-01-11T21:28:27.6922906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4vhf7mo2/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6923069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6923370Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6923495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6923762Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6923893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6924075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbmg9i0xh
2023-01-11T21:28:27.6924268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbmg9i0xh/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6924430Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6924586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6924859Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6924983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6925257Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6925377Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6925559Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwy5292zu
2023-01-11T21:28:27.6925749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwy5292zu/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6925908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6925974Z ok (1.962s)
2023-01-11T21:28:27.6925980Z 
2023-01-11T21:28:27.6926174Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6926247Z Ran 1 test in 1.962s
2023-01-11T21:28:27.6926252Z 
2023-01-11T21:28:27.6926312Z OK
2023-01-11T21:28:27.6926317Z 
2023-01-11T21:28:27.6926391Z Generating XML reports...
2023-01-11T21:28:27.6926716Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212604.xml
2023-01-11T21:28:27.6926985Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6927106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6927384Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6927517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6927698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiz6ci82o
2023-01-11T21:28:27.6927893Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiz6ci82o/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6928122Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6928178Z 
2023-01-11T21:28:27.6928237Z Running tests...
2023-01-11T21:28:27.6928430Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6928668Z   test_allreduce_coalesced_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10012
2023-01-11T21:28:27.6928822Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10013
2023-01-11T21:28:27.6928973Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10014
2023-01-11T21:28:27.6929121Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10015
2023-01-11T21:28:27.6929394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6929513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6929809Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6929946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6930130Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqpl2rm4y
2023-01-11T21:28:27.6930325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqpl2rm4y/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6930597Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6930720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6931196Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6931333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6931517Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0nwfl5gw
2023-01-11T21:28:27.6931701Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0nwfl5gw/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6931862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6932017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6932292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6932414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6932692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6932824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6933006Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpicejnks8
2023-01-11T21:28:27.6933203Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpicejnks8/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6933459Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6933581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6933856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6933988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6934171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_dv2be8r
2023-01-11T21:28:27.6934364Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_dv2be8r/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6934526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6934687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6934792Z ok (2.072s)
2023-01-11T21:28:27.6934798Z 
2023-01-11T21:28:27.6934997Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6935073Z Ran 1 test in 2.072s
2023-01-11T21:28:27.6935078Z 
2023-01-11T21:28:27.6935138Z OK
2023-01-11T21:28:27.6935142Z 
2023-01-11T21:28:27.6935226Z Generating XML reports...
2023-01-11T21:28:27.6935549Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212607.xml
2023-01-11T21:28:27.6935823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6935945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6936224Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6936379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6936564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_w6_2a8k
2023-01-11T21:28:27.6936758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_w6_2a8k/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6936984Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6936989Z 
2023-01-11T21:28:27.6937061Z Running tests...
2023-01-11T21:28:27.6937255Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6937497Z   test_allreduce_coalesced_checks_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10079
2023-01-11T21:28:27.6937651Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10080
2023-01-11T21:28:27.6937789Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10081
2023-01-11T21:28:27.6937946Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10082
2023-01-11T21:28:27.6938221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6938342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6938620Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6938752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6938933Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbr816f10
2023-01-11T21:28:27.6939126Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbr816f10/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6939287Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6939547Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6939671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6939951Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6940085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6940266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3joxb7wb
2023-01-11T21:28:27.6940459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3joxb7wb/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6940726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6940846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6941109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6941272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6941456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0c3qm9p5
2023-01-11T21:28:27.6941651Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0c3qm9p5/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6941812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6942080Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6942200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6942478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6942610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6942876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpistwql8_
2023-01-11T21:28:27.6943080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpistwql8_/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6943241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6943402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6943502Z skip: Need at least 1 CUDA device (1.865s)
2023-01-11T21:28:27.6943508Z 
2023-01-11T21:28:27.6943705Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6943780Z Ran 1 test in 1.866s
2023-01-11T21:28:27.6943785Z 
2023-01-11T21:28:27.6943860Z OK (skipped=1)
2023-01-11T21:28:27.6943865Z 
2023-01-11T21:28:27.6943935Z Generating XML reports...
2023-01-11T21:28:27.6944261Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212611.xml
2023-01-11T21:28:27.6944539Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6944660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6944942Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6945078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6945261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpar97qzdu
2023-01-11T21:28:27.6945455Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpar97qzdu/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6945683Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6945688Z 
2023-01-11T21:28:27.6945747Z Running tests...
2023-01-11T21:28:27.6945941Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6946179Z   test_allreduce_coalesced_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10134
2023-01-11T21:28:27.6946333Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10135
2023-01-11T21:28:27.6946486Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10136
2023-01-11T21:28:27.6946636Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10137
2023-01-11T21:28:27.6946907Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6947028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6947306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6947428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6947612Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1b3g39w0
2023-01-11T21:28:27.6947874Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1b3g39w0/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6948037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6948307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6948429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6948709Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6948842Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6949012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp55q7fplo
2023-01-11T21:28:27.6949234Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp55q7fplo/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6949509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6949632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6949911Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6950044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6950230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpngtgixaz
2023-01-11T21:28:27.6950426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpngtgixaz/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6950589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6950737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6951010Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6951131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6951410Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6951545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6951725Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi16_1djw
2023-01-11T21:28:27.6951915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi16_1djw/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6952074Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6952129Z ok (2.670s)
2023-01-11T21:28:27.6952134Z 
2023-01-11T21:28:27.6952328Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6952406Z Ran 1 test in 2.671s
2023-01-11T21:28:27.6952410Z 
2023-01-11T21:28:27.6952471Z OK
2023-01-11T21:28:27.6952476Z 
2023-01-11T21:28:27.6952558Z Generating XML reports...
2023-01-11T21:28:27.6952879Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212614.xml
2023-01-11T21:28:27.6953148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6953269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6953545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6953666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6953849Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu305oi1m
2023-01-11T21:28:27.6954043Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu305oi1m/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6954297Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6954302Z 
2023-01-11T21:28:27.6954374Z Running tests...
2023-01-11T21:28:27.6954568Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6954793Z   test_allreduce_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10225
2023-01-11T21:28:27.6954946Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10226
2023-01-11T21:28:27.6955086Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10227
2023-01-11T21:28:27.6955236Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10228
2023-01-11T21:28:27.6955509Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6955660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6955938Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6956070Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6956251Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3dg3jliy
2023-01-11T21:28:27.6956443Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3dg3jliy/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6956712Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6956821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6957096Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6957231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6957415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpidlktz08
2023-01-11T21:28:27.6957606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpidlktz08/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6957873Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6957996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6958272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6958393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6958573Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4g5vzk6m
2023-01-11T21:28:27.6958762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4g5vzk6m/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6959036Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6959155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6959431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6959563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6959748Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv7ydcehm
2023-01-11T21:28:27.6959940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv7ydcehm/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6960091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6960251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6960410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6960599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6960665Z ok (2.670s)
2023-01-11T21:28:27.6960670Z 
2023-01-11T21:28:27.6960866Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6960940Z Ran 1 test in 2.671s
2023-01-11T21:28:27.6960944Z 
2023-01-11T21:28:27.6961007Z OK
2023-01-11T21:28:27.6961011Z 
2023-01-11T21:28:27.6961082Z Generating XML reports...
2023-01-11T21:28:27.6961405Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212619.xml
2023-01-11T21:28:27.6961678Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6961799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6962107Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6962243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6962423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp62wj7yid
2023-01-11T21:28:27.6962615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp62wj7yid/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6962842Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6962847Z 
2023-01-11T21:28:27.6962908Z Running tests...
2023-01-11T21:28:27.6963100Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6963330Z   test_allreduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10316
2023-01-11T21:28:27.6963485Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10317
2023-01-11T21:28:27.6963643Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10318
2023-01-11T21:28:27.6963797Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10319
2023-01-11T21:28:27.6964071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6964192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6964458Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6964592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6964775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjdhvs0cb
2023-01-11T21:28:27.6964973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjdhvs0cb/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6965138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6965411Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6965534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6965814Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6965947Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6966116Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplfavf7ct
2023-01-11T21:28:27.6966309Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplfavf7ct/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6966470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6966738Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6966892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6967168Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6967301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6967484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphel7jlq8
2023-01-11T21:28:27.6967663Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphel7jlq8/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6967826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6968098Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6968220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6968526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6968664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6968848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4cnplxrb
2023-01-11T21:28:27.6969041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4cnplxrb/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6969202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6969292Z skip: Need at least 2 CUDA devices (2.070s)
2023-01-11T21:28:27.6969297Z 
2023-01-11T21:28:27.6969492Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6969570Z Ran 1 test in 2.070s
2023-01-11T21:28:27.6969575Z 
2023-01-11T21:28:27.6969646Z OK (skipped=1)
2023-01-11T21:28:27.6969651Z 
2023-01-11T21:28:27.6969735Z Generating XML reports...
2023-01-11T21:28:27.6970061Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212623.xml
2023-01-11T21:28:27.6970335Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6970458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6970733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6971009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6971265Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoupfdf64
2023-01-11T21:28:27.6971462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoupfdf64/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6971694Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6971699Z 
2023-01-11T21:28:27.6971772Z Running tests...
2023-01-11T21:28:27.6971973Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6972200Z   test_barrier_implies_wait (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10371
2023-01-11T21:28:27.6972354Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10372
2023-01-11T21:28:27.6972495Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10373
2023-01-11T21:28:27.6972651Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10374
2023-01-11T21:28:27.6972925Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6973047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6973326Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6973530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6973715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwncyp917
2023-01-11T21:28:27.6973910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwncyp917/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6974073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6974332Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6974458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6974735Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6974870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6975089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgeqd96ne
2023-01-11T21:28:27.6975287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgeqd96ne/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6975554Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6975675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6975939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6976073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6976257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjw2vlx0l
2023-01-11T21:28:27.6976450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjw2vlx0l/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6976611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6976882Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6977002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6977281Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6977415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6977586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp50gng90n
2023-01-11T21:28:27.6977776Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp50gng90n/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6977938Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6978099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6978167Z ok (2.072s)
2023-01-11T21:28:27.6978174Z 
2023-01-11T21:28:27.6978372Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6978446Z Ran 1 test in 2.072s
2023-01-11T21:28:27.6978451Z 
2023-01-11T21:28:27.6978512Z OK
2023-01-11T21:28:27.6978517Z 
2023-01-11T21:28:27.6978588Z Generating XML reports...
2023-01-11T21:28:27.6978908Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212626.xml
2023-01-11T21:28:27.6979180Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6979304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6979578Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6979709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6979895Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeh1ww6nv
2023-01-11T21:28:27.6980118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeh1ww6nv/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6980346Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6980352Z 
2023-01-11T21:28:27.6980412Z Running tests...
2023-01-11T21:28:27.6980606Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6980830Z   test_broadcast_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10438
2023-01-11T21:28:27.6980983Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10439
2023-01-11T21:28:27.6981134Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10440
2023-01-11T21:28:27.6981283Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10441
2023-01-11T21:28:27.6981588Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6981714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6981980Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6982111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6982291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6lnx_lgr
2023-01-11T21:28:27.6982482Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6lnx_lgr/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6982639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6982989Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6983115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6983396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6983529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6983698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv49yu0mz
2023-01-11T21:28:27.6983889Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv49yu0mz/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6984157Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6984281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6984559Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6984693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6984883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg2euk6s_
2023-01-11T21:28:27.6985074Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg2euk6s_/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6985344Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6985452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6985730Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6985863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6986044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2sbfdjem
2023-01-11T21:28:27.6986238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2sbfdjem/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6986402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6986593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6986754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6986808Z ok (2.070s)
2023-01-11T21:28:27.6986813Z 
2023-01-11T21:28:27.6987007Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6987081Z Ran 1 test in 2.071s
2023-01-11T21:28:27.6987086Z 
2023-01-11T21:28:27.6987148Z OK
2023-01-11T21:28:27.6987153Z 
2023-01-11T21:28:27.6987234Z Generating XML reports...
2023-01-11T21:28:27.6987560Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212630.xml
2023-01-11T21:28:27.6987828Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6987981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6988247Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6988379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6988563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm3dmya_r
2023-01-11T21:28:27.6988760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm3dmya_r/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6988985Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6988991Z 
2023-01-11T21:28:27.6989063Z Running tests...
2023-01-11T21:28:27.6989256Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6989482Z   test_broadcast_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10505
2023-01-11T21:28:27.6989640Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10506
2023-01-11T21:28:27.6989780Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10507
2023-01-11T21:28:27.6989930Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10508
2023-01-11T21:28:27.6990198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6990318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6990599Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6990733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6990913Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8fo_lonk
2023-01-11T21:28:27.6991108Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8fo_lonk/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6991269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.6991526Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6991647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6991921Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6992051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6992234Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpegjtu5r7
2023-01-11T21:28:27.6992426Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpegjtu5r7/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6992587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.6992892Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6993000Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6993277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6993407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6993589Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp7y8js22
2023-01-11T21:28:27.6993780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp7y8js22/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6994049Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6994173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6994478Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6994611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6994780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpygmi2kxf
2023-01-11T21:28:27.6994972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpygmi2kxf/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6995132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.6995293Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.6995395Z skip: Need at least 2 CUDA devices (1.970s)
2023-01-11T21:28:27.6995400Z 
2023-01-11T21:28:27.6995598Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6995672Z Ran 1 test in 1.970s
2023-01-11T21:28:27.6995677Z 
2023-01-11T21:28:27.6995748Z OK (skipped=1)
2023-01-11T21:28:27.6995755Z 
2023-01-11T21:28:27.6995828Z Generating XML reports...
2023-01-11T21:28:27.6996154Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212634.xml
2023-01-11T21:28:27.6996422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6996544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6996823Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6996956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6997138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjc7d82fc
2023-01-11T21:28:27.6997331Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjc7d82fc/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6997562Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.6997569Z 
2023-01-11T21:28:27.6997628Z Running tests...
2023-01-11T21:28:27.6997821Z ----------------------------------------------------------------------
2023-01-11T21:28:27.6998044Z   test_broadcast_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10560
2023-01-11T21:28:27.6998197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10561
2023-01-11T21:28:27.6998348Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10562
2023-01-11T21:28:27.6998496Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10563
2023-01-11T21:28:27.6998767Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.6998887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.6999156Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.6999325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.6999508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvnkn21v8
2023-01-11T21:28:27.6999698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvnkn21v8/_remote_module_non_scriptable.py
2023-01-11T21:28:27.6999968Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7000087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7000365Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7000494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7000677Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqui4kwse
2023-01-11T21:28:27.7000891Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqui4kwse/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7001163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7001284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7001561Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7001693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7001876Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpinkoqfd5
2023-01-11T21:28:27.7002070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpinkoqfd5/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7002338Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7002463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7002726Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7002858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7003040Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplbi6f75v
2023-01-11T21:28:27.7003233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplbi6f75v/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7003393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7003553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7003713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7003873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7003929Z ok (1.972s)
2023-01-11T21:28:27.7003934Z 
2023-01-11T21:28:27.7004136Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7004211Z Ran 1 test in 1.972s
2023-01-11T21:28:27.7004215Z 
2023-01-11T21:28:27.7004274Z OK
2023-01-11T21:28:27.7004279Z 
2023-01-11T21:28:27.7004361Z Generating XML reports...
2023-01-11T21:28:27.7004683Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212637.xml
2023-01-11T21:28:27.7004955Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7005074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7005337Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7005475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7005686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwa_vy9v4
2023-01-11T21:28:27.7005880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwa_vy9v4/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7006111Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7006116Z 
2023-01-11T21:28:27.7006187Z Running tests...
2023-01-11T21:28:27.7006378Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7006603Z   test_broadcast_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10627
2023-01-11T21:28:27.7006756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10628
2023-01-11T21:28:27.7006896Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10629
2023-01-11T21:28:27.7007075Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10630
2023-01-11T21:28:27.7007348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7007467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7007746Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7007878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7008059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5l1fj3wc
2023-01-11T21:28:27.7008254Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5l1fj3wc/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7008413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7008672Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7008794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7009075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7009206Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7009388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz_vio50k
2023-01-11T21:28:27.7009578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz_vio50k/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7009845Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7009965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7010227Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7010362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7010543Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6xsioes6
2023-01-11T21:28:27.7010735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6xsioes6/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7011075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7011256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7011527Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7011649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7011927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7012049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7012283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcgfn4wi2
2023-01-11T21:28:27.7012475Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcgfn4wi2/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7012636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7012704Z ok (2.372s)
2023-01-11T21:28:27.7012709Z 
2023-01-11T21:28:27.7012906Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7012981Z Ran 1 test in 2.372s
2023-01-11T21:28:27.7012986Z 
2023-01-11T21:28:27.7013047Z OK
2023-01-11T21:28:27.7013052Z 
2023-01-11T21:28:27.7013120Z Generating XML reports...
2023-01-11T21:28:27.7013443Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212641.xml
2023-01-11T21:28:27.7013753Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7013880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7014162Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7014295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7014483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8qmy9kj9
2023-01-11T21:28:27.7014681Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8qmy9kj9/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7014912Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7014917Z 
2023-01-11T21:28:27.7014979Z Running tests...
2023-01-11T21:28:27.7015175Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7015405Z   test_broadcast_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10718
2023-01-11T21:28:27.7015562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10719
2023-01-11T21:28:27.7015714Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10720
2023-01-11T21:28:27.7015864Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10721
2023-01-11T21:28:27.7016134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7016256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7016522Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7016655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7016838Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbh5p447c
2023-01-11T21:28:27.7017037Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbh5p447c/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7017306Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7017427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7017702Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7017834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7018015Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa2qdos67
2023-01-11T21:28:27.7018194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa2qdos67/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7018355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7018671Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7018793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7019071Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7019204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7019388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5pqv3p53
2023-01-11T21:28:27.7019581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5pqv3p53/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7019731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7019892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7020194Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7020318Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7020595Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7020727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7020910Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6awfc8u2
2023-01-11T21:28:27.7021104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6awfc8u2/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7021265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7021355Z skip: Need at least 2 CUDA devices (1.867s)
2023-01-11T21:28:27.7021360Z 
2023-01-11T21:28:27.7021559Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7021634Z Ran 1 test in 1.867s
2023-01-11T21:28:27.7021644Z 
2023-01-11T21:28:27.7021716Z OK (skipped=1)
2023-01-11T21:28:27.7021721Z 
2023-01-11T21:28:27.7021805Z Generating XML reports...
2023-01-11T21:28:27.7022132Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212645.xml
2023-01-11T21:28:27.7022401Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7022524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7022791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7022997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7023180Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxf_hz6fx
2023-01-11T21:28:27.7023376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxf_hz6fx/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7023610Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7023615Z 
2023-01-11T21:28:27.7023691Z Running tests...
2023-01-11T21:28:27.7023888Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7024108Z   test_empty_tensors (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10773
2023-01-11T21:28:27.7024267Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10774
2023-01-11T21:28:27.7024406Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10775
2023-01-11T21:28:27.7024562Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10776
2023-01-11T21:28:27.7024834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7024959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7025272Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7025404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7025591Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu37vx5vs
2023-01-11T21:28:27.7025785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu37vx5vs/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7025950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7026207Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7026329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7026637Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7026775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7026958Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkh7b5tv_
2023-01-11T21:28:27.7027149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkh7b5tv_/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7027422Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7027544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7027812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7027944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7028127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp76zqeuso
2023-01-11T21:28:27.7028318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp76zqeuso/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7028484Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7028644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7028917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7029038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7029315Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7029435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7029616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnzr8aij
2023-01-11T21:28:27.7029810Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnzr8aij/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7029970Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7030037Z ok (2.078s)
2023-01-11T21:28:27.7030043Z 
2023-01-11T21:28:27.7030242Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7030318Z Ran 1 test in 2.078s
2023-01-11T21:28:27.7030323Z 
2023-01-11T21:28:27.7030383Z OK
2023-01-11T21:28:27.7030387Z 
2023-01-11T21:28:27.7030456Z Generating XML reports...
2023-01-11T21:28:27.7030780Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212648.xml
2023-01-11T21:28:27.7031048Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7031169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7031447Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7031609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7031790Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5qc31g18
2023-01-11T21:28:27.7031981Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5qc31g18/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7032212Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7032217Z 
2023-01-11T21:28:27.7032276Z Running tests...
2023-01-11T21:28:27.7032468Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7032687Z   test_gather_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10840
2023-01-11T21:28:27.7032841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10841
2023-01-11T21:28:27.7033027Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10842
2023-01-11T21:28:27.7033181Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10843
2023-01-11T21:28:27.7033453Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7033573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7033839Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7033970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7034154Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc1pi_q9j
2023-01-11T21:28:27.7034347Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc1pi_q9j/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7034511Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7034783Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7034905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7035181Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7035314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7035485Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ph6byvf
2023-01-11T21:28:27.7035678Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ph6byvf/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7035948Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7036069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7036348Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7036481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7036665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqdh339iy
2023-01-11T21:28:27.7036857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqdh339iy/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7037006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7037166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7037433Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7037556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7037834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7037996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7038177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8msym5dy
2023-01-11T21:28:27.7038369Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8msym5dy/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7038528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7038583Z ok (2.167s)
2023-01-11T21:28:27.7038589Z 
2023-01-11T21:28:27.7038785Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7038861Z Ran 1 test in 2.167s
2023-01-11T21:28:27.7038866Z 
2023-01-11T21:28:27.7038925Z OK
2023-01-11T21:28:27.7038930Z 
2023-01-11T21:28:27.7039014Z Generating XML reports...
2023-01-11T21:28:27.7039366Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212652.xml
2023-01-11T21:28:27.7039641Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7039762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7040026Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7040157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7040339Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ci255d_
2023-01-11T21:28:27.7040531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ci255d_/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7040756Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7040761Z 
2023-01-11T21:28:27.7040835Z Running tests...
2023-01-11T21:28:27.7041029Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7041256Z   test_gather_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10907
2023-01-11T21:28:27.7041413Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10908
2023-01-11T21:28:27.7041555Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10909
2023-01-11T21:28:27.7041705Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10910
2023-01-11T21:28:27.7041975Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7042095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7042375Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7042507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7042693Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnsza364r
2023-01-11T21:28:27.7042886Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnsza364r/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7043139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7043259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7043533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7043666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7043848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptnt686xb
2023-01-11T21:28:27.7044040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptnt686xb/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7044311Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7044464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7044741Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7044861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7045042Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxaho0jy7
2023-01-11T21:28:27.7045233Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxaho0jy7/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7045396Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7045553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7045773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7046045Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7046166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7046431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7046563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7046747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp649mw6li
2023-01-11T21:28:27.7046940Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp649mw6li/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7047102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7047207Z skip: Need at least 2 CUDA devices (1.968s)
2023-01-11T21:28:27.7047212Z 
2023-01-11T21:28:27.7047411Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7047486Z Ran 1 test in 1.968s
2023-01-11T21:28:27.7047491Z 
2023-01-11T21:28:27.7047563Z OK (skipped=1)
2023-01-11T21:28:27.7047567Z 
2023-01-11T21:28:27.7047637Z Generating XML reports...
2023-01-11T21:28:27.7047965Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212656.xml
2023-01-11T21:28:27.7048235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7048358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7048634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7048767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7048954Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb3xiqqx2
2023-01-11T21:28:27.7049152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb3xiqqx2/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7049380Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7049386Z 
2023-01-11T21:28:27.7049445Z Running tests...
2023-01-11T21:28:27.7049638Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7049859Z   test_gather_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10962
2023-01-11T21:28:27.7050013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10963
2023-01-11T21:28:27.7050168Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 10964
2023-01-11T21:28:27.7050318Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 10965
2023-01-11T21:28:27.7050593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7050755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7051261Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7051397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7051585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpak80fzf5
2023-01-11T21:28:27.7051780Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpak80fzf5/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7051941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7052211Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7052332Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7052661Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7052794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7052965Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzo3shpne
2023-01-11T21:28:27.7053160Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzo3shpne/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7053321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7053594Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7053716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7053993Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7054132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7054317Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc92q5_2i
2023-01-11T21:28:27.7054493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc92q5_2i/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7054763Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7054885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7055163Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7055296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7055479Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp63tga79j
2023-01-11T21:28:27.7055646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7055840Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp63tga79j/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7056002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7056056Z ok (2.072s)
2023-01-11T21:28:27.7056062Z 
2023-01-11T21:28:27.7056259Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7056334Z Ran 1 test in 2.073s
2023-01-11T21:28:27.7056339Z 
2023-01-11T21:28:27.7056400Z OK
2023-01-11T21:28:27.7056404Z 
2023-01-11T21:28:27.7056486Z Generating XML reports...
2023-01-11T21:28:27.7056807Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212659.xml
2023-01-11T21:28:27.7057075Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7057197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7057502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7057636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7057816Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxn1r_z5y
2023-01-11T21:28:27.7058008Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxn1r_z5y/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7058240Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7058245Z 
2023-01-11T21:28:27.7058316Z Running tests...
2023-01-11T21:28:27.7058510Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7058748Z   test_gather_noncontiguous_input (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11029
2023-01-11T21:28:27.7058931Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11030
2023-01-11T21:28:27.7059076Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11031
2023-01-11T21:28:27.7059228Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11032
2023-01-11T21:28:27.7059501Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7059622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7059903Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7060037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7060222Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn47r3uhc
2023-01-11T21:28:27.7060417Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn47r3uhc/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7060570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7060838Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7060960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7061235Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7061367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7061548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jtewrgs
2023-01-11T21:28:27.7061743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jtewrgs/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7061903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7062174Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7062285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7062566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7062699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7062951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu9p2pkvf
2023-01-11T21:28:27.7063148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu9p2pkvf/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7063312Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7063587Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7063711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7064042Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7064161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7064343Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3oin0jji
2023-01-11T21:28:27.7064537Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3oin0jji/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7064700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7064767Z ok (2.071s)
2023-01-11T21:28:27.7064773Z 
2023-01-11T21:28:27.7064969Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7065043Z Ran 1 test in 2.071s
2023-01-11T21:28:27.7065047Z 
2023-01-11T21:28:27.7065107Z OK
2023-01-11T21:28:27.7065112Z 
2023-01-11T21:28:27.7065181Z Generating XML reports...
2023-01-11T21:28:27.7065538Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212703.xml
2023-01-11T21:28:27.7065812Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7065933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7066210Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7066346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7066529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_5n62wmi
2023-01-11T21:28:27.7066722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_5n62wmi/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7066937Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7066958Z 
2023-01-11T21:28:27.7067019Z Running tests...
2023-01-11T21:28:27.7067215Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7067434Z   test_gather_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11096
2023-01-11T21:28:27.7067590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11097
2023-01-11T21:28:27.7067742Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11098
2023-01-11T21:28:27.7067891Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11099
2023-01-11T21:28:27.7068161Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7068282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7068545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7068680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7068864Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpji6fn01e
2023-01-11T21:28:27.7069059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpji6fn01e/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7069220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7069489Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7069611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7069888Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7070019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7070191Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2e77el4z
2023-01-11T21:28:27.7070411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2e77el4z/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7070573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7070840Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7070960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7071239Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7071370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7071552Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsztazg3j
2023-01-11T21:28:27.7071766Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsztazg3j/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7071931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7072205Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7072327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7072605Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7072738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7072921Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwm1m9iwl
2023-01-11T21:28:27.7073115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwm1m9iwl/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7073276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7073337Z ok (3.668s)
2023-01-11T21:28:27.7073343Z 
2023-01-11T21:28:27.7073540Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7073616Z Ran 1 test in 3.669s
2023-01-11T21:28:27.7073621Z 
2023-01-11T21:28:27.7073683Z OK
2023-01-11T21:28:27.7073687Z 
2023-01-11T21:28:27.7073772Z Generating XML reports...
2023-01-11T21:28:27.7074095Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212707.xml
2023-01-11T21:28:27.7074366Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7074486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7074750Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7074884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7075070Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp33l8jeye
2023-01-11T21:28:27.7075266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp33l8jeye/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7075500Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7075506Z 
2023-01-11T21:28:27.7075579Z Running tests...
2023-01-11T21:28:27.7075772Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7076005Z   test_gather_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11187
2023-01-11T21:28:27.7076162Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11188
2023-01-11T21:28:27.7076301Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11189
2023-01-11T21:28:27.7076455Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11190
2023-01-11T21:28:27.7076761Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7076882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7077159Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7077292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7077477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9vzg3tlz
2023-01-11T21:28:27.7077671Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9vzg3tlz/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7077930Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7078053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7078361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7078498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7078676Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqce_iahg
2023-01-11T21:28:27.7078866Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqce_iahg/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7079027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7079190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7079461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7079571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7079852Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7079988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7080173Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx_762npa
2023-01-11T21:28:27.7080365Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx_762npa/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7080634Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7080757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7081034Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7081154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7081334Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiilzqq3f
2023-01-11T21:28:27.7081531Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiilzqq3f/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7081692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7081849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7081952Z skip: Need at least 2 CUDA devices (1.867s)
2023-01-11T21:28:27.7081958Z 
2023-01-11T21:28:27.7082154Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7082227Z Ran 1 test in 1.868s
2023-01-11T21:28:27.7082232Z 
2023-01-11T21:28:27.7082302Z OK (skipped=1)
2023-01-11T21:28:27.7082307Z 
2023-01-11T21:28:27.7082377Z Generating XML reports...
2023-01-11T21:28:27.7082698Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212712.xml
2023-01-11T21:28:27.7082973Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7083129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7083407Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7083540Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7083724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwprsirui
2023-01-11T21:28:27.7083917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwprsirui/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7084133Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7084149Z 
2023-01-11T21:28:27.7084208Z Running tests...
2023-01-11T21:28:27.7084404Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7084682Z   test_multi_device_constructor (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11242
2023-01-11T21:28:27.7084841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11243
2023-01-11T21:28:27.7084995Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11244
2023-01-11T21:28:27.7085143Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11245
2023-01-11T21:28:27.7085414Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7085537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7085803Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7085935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7086120Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcc1e3g8a
2023-01-11T21:28:27.7086322Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcc1e3g8a/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7086591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7086713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7086991Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7087124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7087308Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprgjn6q1g
2023-01-11T21:28:27.7087489Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprgjn6q1g/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7087651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7087923Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7088048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7088325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7088458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7088640Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ji6ljg2
2023-01-11T21:28:27.7088831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ji6ljg2/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7088980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7089248Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7089372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7089686Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7089818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7089997Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq939znrn
2023-01-11T21:28:27.7090189Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq939znrn/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7090348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7090507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7090562Z ok (2.072s)
2023-01-11T21:28:27.7090568Z 
2023-01-11T21:28:27.7090765Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7090979Z Ran 1 test in 2.072s
2023-01-11T21:28:27.7090988Z 
2023-01-11T21:28:27.7091083Z OK
2023-01-11T21:28:27.7091150Z 
2023-01-11T21:28:27.7091237Z Generating XML reports...
2023-01-11T21:28:27.7091565Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212715.xml
2023-01-11T21:28:27.7091834Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7091958Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7092218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7092352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7092538Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvvjvt8vx
2023-01-11T21:28:27.7092733Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvvjvt8vx/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7092967Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7092973Z 
2023-01-11T21:28:27.7093045Z Running tests...
2023-01-11T21:28:27.7093238Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7093461Z   test_reduce_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11313
2023-01-11T21:28:27.7093617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11314
2023-01-11T21:28:27.7093757Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11315
2023-01-11T21:28:27.7093913Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11316
2023-01-11T21:28:27.7094186Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7094309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7094592Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7094725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7094907Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp98gua1r8
2023-01-11T21:28:27.7095100Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp98gua1r8/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7095248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7095516Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7095637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7095914Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7096049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7096268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8iwb53vl
2023-01-11T21:28:27.7096462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8iwb53vl/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7096623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7096893Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7097002Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7097279Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7097411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7097595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ytdn120
2023-01-11T21:28:27.7097818Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ytdn120/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7097981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7098252Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7098372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7098639Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7098773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7098956Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr4fzufl6
2023-01-11T21:28:27.7099150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr4fzufl6/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7099316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7099384Z ok (2.169s)
2023-01-11T21:28:27.7099389Z 
2023-01-11T21:28:27.7099586Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7099661Z Ran 1 test in 2.170s
2023-01-11T21:28:27.7099666Z 
2023-01-11T21:28:27.7099725Z OK
2023-01-11T21:28:27.7099730Z 
2023-01-11T21:28:27.7099800Z Generating XML reports...
2023-01-11T21:28:27.7100124Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212719.xml
2023-01-11T21:28:27.7100394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7100515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7100791Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7100927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7101110Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxn_g9_gf
2023-01-11T21:28:27.7101302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxn_g9_gf/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7101516Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7101533Z 
2023-01-11T21:28:27.7101592Z Running tests...
2023-01-11T21:28:27.7101787Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7102011Z   test_reduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11380
2023-01-11T21:28:27.7102164Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11381
2023-01-11T21:28:27.7102317Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11382
2023-01-11T21:28:27.7102505Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11383
2023-01-11T21:28:27.7102779Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7102969Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7103243Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7103375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7103558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpirzkxp4p
2023-01-11T21:28:27.7103753Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpirzkxp4p/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7103914Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7104221Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7104346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7104631Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7104751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7104935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4qipslsb
2023-01-11T21:28:27.7105128Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4qipslsb/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7105394Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7105516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7105797Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7105930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7106112Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk2ywktzw
2023-01-11T21:28:27.7106303Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk2ywktzw/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7106452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7106722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7106842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7107116Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7107248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7107434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0tqa6lw
2023-01-11T21:28:27.7107624Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0tqa6lw/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7107783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7107943Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7108033Z skip: Need at least 2 CUDA devices (1.969s)
2023-01-11T21:28:27.7108038Z 
2023-01-11T21:28:27.7108233Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7108310Z Ran 1 test in 1.970s
2023-01-11T21:28:27.7108315Z 
2023-01-11T21:28:27.7108385Z OK (skipped=1)
2023-01-11T21:28:27.7108389Z 
2023-01-11T21:28:27.7108472Z Generating XML reports...
2023-01-11T21:28:27.7108794Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212723.xml
2023-01-11T21:28:27.7109109Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7109229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7109494Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7109626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7109809Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpddz0w_4l
2023-01-11T21:28:27.7110000Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpddz0w_4l/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7110228Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7110234Z 
2023-01-11T21:28:27.7110305Z Running tests...
2023-01-11T21:28:27.7110499Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7110754Z   test_reduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11435
2023-01-11T21:28:27.7110910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11436
2023-01-11T21:28:27.7111050Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11437
2023-01-11T21:28:27.7111200Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11438
2023-01-11T21:28:27.7111471Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7111592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7111868Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7112001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7112190Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpan3h468f
2023-01-11T21:28:27.7112382Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpan3h468f/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7112531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7112800Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7112921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7113198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7113331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7113514Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp99ci5thr
2023-01-11T21:28:27.7113710Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp99ci5thr/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7113979Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7114098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7114361Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7114491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7114674Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdivotnem
2023-01-11T21:28:27.7114868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdivotnem/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7115134Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7115255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7115572Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7115704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7115875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf0pakg_f
2023-01-11T21:28:27.7116068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf0pakg_f/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7116231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7116391Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7116550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7116616Z ok (2.074s)
2023-01-11T21:28:27.7116621Z 
2023-01-11T21:28:27.7116819Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7116929Z Ran 1 test in 2.074s
2023-01-11T21:28:27.7116935Z 
2023-01-11T21:28:27.7116995Z OK
2023-01-11T21:28:27.7117000Z 
2023-01-11T21:28:27.7117071Z Generating XML reports...
2023-01-11T21:28:27.7117393Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212726.xml
2023-01-11T21:28:27.7117665Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7117785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7118062Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7118196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7118378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpacwlv4ou
2023-01-11T21:28:27.7118576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpacwlv4ou/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7118793Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7118810Z 
2023-01-11T21:28:27.7118868Z Running tests...
2023-01-11T21:28:27.7119062Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7119281Z   test_reduce_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11502
2023-01-11T21:28:27.7119434Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11503
2023-01-11T21:28:27.7119587Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11504
2023-01-11T21:28:27.7119736Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11505
2023-01-11T21:28:27.7120007Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7120131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7120396Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7120526Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7120709Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeiwrbxy5
2023-01-11T21:28:27.7120901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeiwrbxy5/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7121063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7121329Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7121450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7121733Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7121885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7122067Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxriil_u
2023-01-11T21:28:27.7122261Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxriil_u/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7122422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7122691Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7122811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7123089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7123220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7123436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa8q7170l
2023-01-11T21:28:27.7123692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7123812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7124089Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7124222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7124407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphd5a3pfp
2023-01-11T21:28:27.7124599Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa8q7170l/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7124788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphd5a3pfp/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7124952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7125102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7125171Z ok (3.374s)
2023-01-11T21:28:27.7125176Z 
2023-01-11T21:28:27.7125373Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7125448Z Ran 1 test in 3.374s
2023-01-11T21:28:27.7125453Z 
2023-01-11T21:28:27.7125515Z OK
2023-01-11T21:28:27.7125520Z 
2023-01-11T21:28:27.7125603Z Generating XML reports...
2023-01-11T21:28:27.7125925Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212730.xml
2023-01-11T21:28:27.7126198Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7126323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7126591Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7126728Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7126914Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppyeqxhpk
2023-01-11T21:28:27.7127112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppyeqxhpk/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7127342Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7127347Z 
2023-01-11T21:28:27.7127420Z Running tests...
2023-01-11T21:28:27.7127615Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7127843Z   test_reduce_stress_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11593
2023-01-11T21:28:27.7127999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11594
2023-01-11T21:28:27.7128175Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11595
2023-01-11T21:28:27.7128329Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11596
2023-01-11T21:28:27.7128601Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7128724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7129002Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7129135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7129321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuenjbu8n
2023-01-11T21:28:27.7129517Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuenjbu8n/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7129694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7129970Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7130093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7130372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7130506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7130687Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_us1fsvx
2023-01-11T21:28:27.7131035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_us1fsvx/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7131265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7131544Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7131656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7131935Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7132071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7132254Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6v4vi7_t
2023-01-11T21:28:27.7132524Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7132649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7132927Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7133062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7133236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvf57oi0
2023-01-11T21:28:27.7133434Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6v4vi7_t/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7133622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvf57oi0/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7133785Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7133946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7134051Z skip: Need at least 2 CUDA devices (1.864s)
2023-01-11T21:28:27.7134056Z 
2023-01-11T21:28:27.7134255Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7134330Z Ran 1 test in 1.865s
2023-01-11T21:28:27.7134335Z 
2023-01-11T21:28:27.7134408Z OK (skipped=1)
2023-01-11T21:28:27.7134413Z 
2023-01-11T21:28:27.7134485Z Generating XML reports...
2023-01-11T21:28:27.7134811Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212735.xml
2023-01-11T21:28:27.7135139Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7135265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7135546Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7135680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7135863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_hrt4ws
2023-01-11T21:28:27.7136058Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_hrt4ws/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7136274Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7136295Z 
2023-01-11T21:28:27.7136359Z Running tests...
2023-01-11T21:28:27.7136595Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7136816Z   test_round_robin (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11648
2023-01-11T21:28:27.7136971Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11649
2023-01-11T21:28:27.7137123Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11650
2023-01-11T21:28:27.7137277Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11651
2023-01-11T21:28:27.7137550Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7137672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7137940Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7138075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7138257Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2x4c633a
2023-01-11T21:28:27.7138450Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2x4c633a/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7138721Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7138841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7139119Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7139250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7139421Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv459yupb
2023-01-11T21:28:27.7139613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv459yupb/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7139779Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7139941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7140209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7140330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7140610Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7140741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7140924Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0e4laazn
2023-01-11T21:28:27.7141104Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0e4laazn/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7141296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7141565Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7141686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7141964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7142094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7142275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6orziego
2023-01-11T21:28:27.7142469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6orziego/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7142616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7142914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.7143095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:28:27.7143269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.7143572Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:28:27.7143741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:28:27.7144035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:28:27.7144332Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:28:27.7144624Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:28:27.7144797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:28:27.7144957Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:28:27.7145131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:28:27.7145421Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:28:27.7145593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T21:28:27.7145888Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:28:27.7146177Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:28:27.7146471Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:28:27.7146640Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:28:27.7146810Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:28:27.7146968Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T21:28:27.7147134Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T21:28:27.7147425Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:28:27.7147719Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:28:27.7148010Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:28:27.7148373Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:28:27.7148801Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2023-01-11T21:28:27.7149212Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2023-01-11T21:28:27.7149647Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2023-01-11T21:28:27.7150051Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2023-01-11T21:28:27.7150119Z ok (2.264s)
2023-01-11T21:28:27.7150124Z 
2023-01-11T21:28:27.7150328Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7150404Z Ran 1 test in 2.265s
2023-01-11T21:28:27.7150409Z 
2023-01-11T21:28:27.7150457Z OK
2023-01-11T21:28:27.7150462Z 
2023-01-11T21:28:27.7150546Z Generating XML reports...
2023-01-11T21:28:27.7150873Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212738.xml
2023-01-11T21:28:27.7151149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7151273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7151555Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7151691Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7151877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphvc8siad
2023-01-11T21:28:27.7152061Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphvc8siad/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7152289Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7152295Z 
2023-01-11T21:28:27.7152365Z Running tests...
2023-01-11T21:28:27.7152565Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7152800Z   test_round_robin_create_destroy (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11747
2023-01-11T21:28:27.7152953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11748
2023-01-11T21:28:27.7153105Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11749
2023-01-11T21:28:27.7153256Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11750
2023-01-11T21:28:27.7153528Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7153637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7153917Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7154054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7154285Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcsezwwhy
2023-01-11T21:28:27.7154479Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcsezwwhy/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7154748Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7154870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7155149Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7155269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7155452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfqn8f9z0
2023-01-11T21:28:27.7155645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfqn8f9z0/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7155941Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7156063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7156341Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7156473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7156660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqc7lpybk
2023-01-11T21:28:27.7156857Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqc7lpybk/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7157007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7157170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7157335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7157604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7157726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7158004Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7158137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7158319Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptur511bl
2023-01-11T21:28:27.7158497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptur511bl/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7158657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7158832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.7159008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 2
2023-01-11T21:28:27.7159175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:28:27.7159345Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 3
2023-01-11T21:28:27.7159646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:28:27.7159939Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:28:27.7160228Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:28:27.7160513Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:1 with 4 nodes.
2023-01-11T21:28:27.7160704Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:28:27.7160874Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:28:27.7161044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 3
2023-01-11T21:28:27.7161213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 2
2023-01-11T21:28:27.7161510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:28:27.7161800Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:28:27.7162091Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:28:27.7162400Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:2 with 4 nodes.
2023-01-11T21:28:27.7162571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:28:27.7162726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:28:27.7162894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 3
2023-01-11T21:28:27.7163061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 2
2023-01-11T21:28:27.7163354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:28:27.7163644Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:28:27.7163933Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:28:27.7164223Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:3 with 4 nodes.
2023-01-11T21:28:27.7164640Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2023-01-11T21:28:27.7165049Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2023-01-11T21:28:27.7165454Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2023-01-11T21:28:27.7165853Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2023-01-11T21:28:27.7166029Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T21:28:27.7166203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 2
2023-01-11T21:28:27.7166361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 3
2023-01-11T21:28:27.7166660Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:28:27.7166859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T21:28:27.7167152Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:28:27.7167445Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:28:27.7167736Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 4 nodes.
2023-01-11T21:28:27.7167908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T21:28:27.7168076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T21:28:27.7168245Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 3
2023-01-11T21:28:27.7168438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 2
2023-01-11T21:28:27.7168731Z INFO:torch.distributed.distributed_c10d:Rank 2: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:28:27.7169024Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:28:27.7169313Z INFO:torch.distributed.distributed_c10d:Rank 3: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:28:27.7169600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 4 nodes.
2023-01-11T21:28:27.7170011Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2023-01-11T21:28:27.7170413Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2023-01-11T21:28:27.7170933Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2023-01-11T21:28:27.7171399Z [W ProcessGroupRoundRobin.cpp:12] Warning: ProcessGroupRoundRobin is deprecated and scheduled to be removed after this current release (1.13). Please file an issue on https://github.com/pytorch/pytorch/issues if there are any concerns or issues with this deprecation. (function ProcessGroupRoundRobin)
2023-01-11T21:28:27.7171472Z ok (2.368s)
2023-01-11T21:28:27.7171477Z 
2023-01-11T21:28:27.7171678Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7171754Z Ran 1 test in 2.368s
2023-01-11T21:28:27.7171759Z 
2023-01-11T21:28:27.7171806Z OK
2023-01-11T21:28:27.7171829Z 
2023-01-11T21:28:27.7171900Z Generating XML reports...
2023-01-11T21:28:27.7172224Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212742.xml
2023-01-11T21:28:27.7172496Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7172620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7172899Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7173033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7173273Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp7n_9qvo
2023-01-11T21:28:27.7173471Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp7n_9qvo/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7173687Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7173705Z 
2023-01-11T21:28:27.7173764Z Running tests...
2023-01-11T21:28:27.7173960Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7174182Z   test_scatter_basics (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11878
2023-01-11T21:28:27.7174340Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11879
2023-01-11T21:28:27.7174493Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11880
2023-01-11T21:28:27.7174647Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11881
2023-01-11T21:28:27.7174961Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7175086Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7175357Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7175493Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7175678Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn284ozex
2023-01-11T21:28:27.7175873Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn284ozex/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7176033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7176302Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7176428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7176706Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7176826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7177011Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwpaj3r_e
2023-01-11T21:28:27.7177207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwpaj3r_e/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7177372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7177644Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7177766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7178046Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7178179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7178362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ku11xzp
2023-01-11T21:28:27.7178543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ku11xzp/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7178703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7178974Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7179095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7179373Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7179506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7179723Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj5odubha
2023-01-11T21:28:27.7179916Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj5odubha/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7180065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7180132Z ok (1.970s)
2023-01-11T21:28:27.7180138Z 
2023-01-11T21:28:27.7180332Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7180406Z Ran 1 test in 1.971s
2023-01-11T21:28:27.7180411Z 
2023-01-11T21:28:27.7180471Z OK
2023-01-11T21:28:27.7180476Z 
2023-01-11T21:28:27.7180558Z Generating XML reports...
2023-01-11T21:28:27.7180880Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212746.xml
2023-01-11T21:28:27.7181148Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7181298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7181566Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7181702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7181884Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp22umypyj
2023-01-11T21:28:27.7182077Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp22umypyj/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7182305Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7182310Z 
2023-01-11T21:28:27.7182381Z Running tests...
2023-01-11T21:28:27.7182576Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7182862Z   test_scatter_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11945
2023-01-11T21:28:27.7183013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11946
2023-01-11T21:28:27.7183168Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 11947
2023-01-11T21:28:27.7183320Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 11948
2023-01-11T21:28:27.7183593Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7183718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7183998Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7184132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7184320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf12lnghb
2023-01-11T21:28:27.7184518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf12lnghb/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7184669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7184947Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7185070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7185347Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7185481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7185665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4bsxjyb3
2023-01-11T21:28:27.7185861Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4bsxjyb3/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7186133Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7186276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7186553Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7186687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7186875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7qhlak0f
2023-01-11T21:28:27.7187067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7qhlak0f/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7187229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7187495Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7187618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7187926Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7188047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7188231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb44a9ki2
2023-01-11T21:28:27.7188420Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb44a9ki2/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7188582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7188742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7188846Z skip: Need at least 2 CUDA devices (1.969s)
2023-01-11T21:28:27.7188852Z 
2023-01-11T21:28:27.7189050Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7189126Z Ran 1 test in 1.969s
2023-01-11T21:28:27.7189131Z 
2023-01-11T21:28:27.7189194Z OK (skipped=1)
2023-01-11T21:28:27.7189213Z 
2023-01-11T21:28:27.7189284Z Generating XML reports...
2023-01-11T21:28:27.7189610Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212750.xml
2023-01-11T21:28:27.7189885Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7190006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7190283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7190418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7190604Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7kecbtcz
2023-01-11T21:28:27.7190800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7kecbtcz/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7191018Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7191037Z 
2023-01-11T21:28:27.7191097Z Running tests...
2023-01-11T21:28:27.7191296Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7191518Z   test_scatter_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12000
2023-01-11T21:28:27.7191674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12001
2023-01-11T21:28:27.7191830Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12002
2023-01-11T21:28:27.7191981Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12003
2023-01-11T21:28:27.7192253Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7192377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7192673Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7192809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7192993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpof91winp
2023-01-11T21:28:27.7193187Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpof91winp/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7193348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7193618Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7193742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7194020Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7194175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7194362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdwvsrnmr
2023-01-11T21:28:27.7194555Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdwvsrnmr/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7194830Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7194952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7195231Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7195362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7195544Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpicp6vcz4
2023-01-11T21:28:27.7195738Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpicp6vcz4/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7195890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7196052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7196325Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7196445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7196722Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7196854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7197037Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp90uicbmt
2023-01-11T21:28:27.7197231Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp90uicbmt/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7197384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7197450Z ok (1.976s)
2023-01-11T21:28:27.7197455Z 
2023-01-11T21:28:27.7197651Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7197727Z Ran 1 test in 1.976s
2023-01-11T21:28:27.7197732Z 
2023-01-11T21:28:27.7197794Z OK
2023-01-11T21:28:27.7197798Z 
2023-01-11T21:28:27.7197881Z Generating XML reports...
2023-01-11T21:28:27.7198205Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212753.xml
2023-01-11T21:28:27.7198473Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7198594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7198856Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7199033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7199216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5vcsejbo
2023-01-11T21:28:27.7199412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5vcsejbo/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7199640Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7199645Z 
2023-01-11T21:28:27.7199717Z Running tests...
2023-01-11T21:28:27.7199911Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7200128Z   test_scatter_stress (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12067
2023-01-11T21:28:27.7200270Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12068
2023-01-11T21:28:27.7200421Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12069
2023-01-11T21:28:27.7200604Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12070
2023-01-11T21:28:27.7200878Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7200999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7201277Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7201412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7201599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptgkwavvf
2023-01-11T21:28:27.7201795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptgkwavvf/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7201945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7202214Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7202338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7202613Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7202744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7202923Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_811b07r
2023-01-11T21:28:27.7203112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_811b07r/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7203275Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7203533Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7203653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7203936Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7204068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7204253Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdkr6_uq9
2023-01-11T21:28:27.7204446Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdkr6_uq9/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7204605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7204876Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7204998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7205262Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7205428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7205610Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv82dedq2
2023-01-11T21:28:27.7205800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv82dedq2/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7205960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7206029Z ok (3.772s)
2023-01-11T21:28:27.7206034Z 
2023-01-11T21:28:27.7206230Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7206303Z Ran 1 test in 3.772s
2023-01-11T21:28:27.7206309Z 
2023-01-11T21:28:27.7206356Z OK
2023-01-11T21:28:27.7206372Z 
2023-01-11T21:28:27.7206442Z Generating XML reports...
2023-01-11T21:28:27.7206763Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212757.xml
2023-01-11T21:28:27.7207061Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7207184Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7207461Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7207593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7207776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkoax_0h2
2023-01-11T21:28:27.7207966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkoax_0h2/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7208183Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7208188Z 
2023-01-11T21:28:27.7208259Z Running tests...
2023-01-11T21:28:27.7208452Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7208685Z   test_scatter_stress_cuda (__main__.ProcessGroupGlooTest) ... skip: Test is flaky, see https://github.com/pytorch/pytorch/issues/15963 (0.001s)
2023-01-11T21:28:27.7208690Z 
2023-01-11T21:28:27.7208878Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7208952Z Ran 1 test in 0.001s
2023-01-11T21:28:27.7208957Z 
2023-01-11T21:28:27.7209026Z OK (skipped=1)
2023-01-11T21:28:27.7209031Z 
2023-01-11T21:28:27.7209113Z Generating XML reports...
2023-01-11T21:28:27.7209434Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212802.xml
2023-01-11T21:28:27.7209692Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7209812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7210090Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7210227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7210410Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_9sas3vr
2023-01-11T21:28:27.7210601Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_9sas3vr/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7210947Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7210953Z 
2023-01-11T21:28:27.7211066Z Running tests...
2023-01-11T21:28:27.7211291Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7211516Z   test_send_recv_all_to_all (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12168
2023-01-11T21:28:27.7211672Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12169
2023-01-11T21:28:27.7211825Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12170
2023-01-11T21:28:27.7212032Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12171
2023-01-11T21:28:27.7212307Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7212430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7212710Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7212844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7213015Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph2p4h_93
2023-01-11T21:28:27.7213283Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7213404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7213719Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7213855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7214045Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbmsghluv
2023-01-11T21:28:27.7214242Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph2p4h_93/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7214435Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbmsghluv/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7214587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7214742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7215012Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7215135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7215413Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7215547Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7215729Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4jmwi80h
2023-01-11T21:28:27.7215920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4jmwi80h/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7216191Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7216300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7216576Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7216706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7216891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp44afazjl
2023-01-11T21:28:27.7217083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp44afazjl/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7217242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7217401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7217468Z ok (1.971s)
2023-01-11T21:28:27.7217473Z 
2023-01-11T21:28:27.7217668Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7217729Z Ran 1 test in 1.972s
2023-01-11T21:28:27.7217735Z 
2023-01-11T21:28:27.7217794Z OK
2023-01-11T21:28:27.7217800Z 
2023-01-11T21:28:27.7217882Z Generating XML reports...
2023-01-11T21:28:27.7218201Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212804.xml
2023-01-11T21:28:27.7218502Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7218622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7218900Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7219036Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7219205Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuvr0274y
2023-01-11T21:28:27.7219394Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuvr0274y/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7219623Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7219628Z 
2023-01-11T21:28:27.7219699Z Running tests...
2023-01-11T21:28:27.7219893Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7220121Z   test_sparse_allreduce_basics (__main__.ProcessGroupGlooTest) ... skip: intermittent failures on Windows, in CI (0.000s)
2023-01-11T21:28:27.7220127Z 
2023-01-11T21:28:27.7220320Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7220395Z Ran 1 test in 0.000s
2023-01-11T21:28:27.7220400Z 
2023-01-11T21:28:27.7220470Z OK (skipped=1)
2023-01-11T21:28:27.7220474Z 
2023-01-11T21:28:27.7220545Z Generating XML reports...
2023-01-11T21:28:27.7220866Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212807.xml
2023-01-11T21:28:27.7221135Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7221255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7221534Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7221671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7221855Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8rkxiuka
2023-01-11T21:28:27.7222049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8rkxiuka/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7222262Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7222279Z 
2023-01-11T21:28:27.7222337Z Running tests...
2023-01-11T21:28:27.7222529Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7222767Z   test_sparse_allreduce_basics_cuda (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12245
2023-01-11T21:28:27.7222987Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12246
2023-01-11T21:28:27.7223143Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12247
2023-01-11T21:28:27.7223301Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12248
2023-01-11T21:28:27.7223575Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7223697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7223964Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7224098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7224283Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6o38r49k
2023-01-11T21:28:27.7224478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6o38r49k/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7224641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7224915Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7225071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7225352Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7225484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7225655Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuyd1778w
2023-01-11T21:28:27.7225847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuyd1778w/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7226117Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7226238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7226545Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7226680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7226862Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_qy_ofh8
2023-01-11T21:28:27.7227051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_qy_ofh8/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7227201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7227361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7227633Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7227753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7228029Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7228165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7228348Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpshneqauf
2023-01-11T21:28:27.7228541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpshneqauf/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7228700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7228791Z skip: Need at least 2 CUDA devices (1.981s)
2023-01-11T21:28:27.7228796Z 
2023-01-11T21:28:27.7228994Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7229069Z Ran 1 test in 1.981s
2023-01-11T21:28:27.7229075Z 
2023-01-11T21:28:27.7229144Z OK (skipped=1)
2023-01-11T21:28:27.7229149Z 
2023-01-11T21:28:27.7229231Z Generating XML reports...
2023-01-11T21:28:27.7229558Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212809.xml
2023-01-11T21:28:27.7229832Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7229952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7230218Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7230350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7230534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjz8db2p4
2023-01-11T21:28:27.7230726Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjz8db2p4/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7230956Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7230961Z 
2023-01-11T21:28:27.7231033Z Running tests...
2023-01-11T21:28:27.7231230Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7231491Z   test_sparse_allreduce_checks (__main__.ProcessGroupGlooTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12300
2023-01-11T21:28:27.7231645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12301
2023-01-11T21:28:27.7231786Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 12302
2023-01-11T21:28:27.7231938Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 12303
2023-01-11T21:28:27.7232209Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7232328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7232604Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7232735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7232949Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7sgf3yge
2023-01-11T21:28:27.7233143Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7sgf3yge/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7233292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T21:28:27.7233558Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7233679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7233958Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7234090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7234271Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoviusdo0
2023-01-11T21:28:27.7234541Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7234664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7234939Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7235058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7235239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp56footmk
2023-01-11T21:28:27.7235431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoviusdo0/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7235621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp56footmk/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7235889Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7236014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7236292Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7236424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7236594Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp8vh_jd1
2023-01-11T21:28:27.7236781Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp8vh_jd1/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7236941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:28:27.7237100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T21:28:27.7237259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:28:27.7237324Z ok (1.972s)
2023-01-11T21:28:27.7237329Z 
2023-01-11T21:28:27.7237529Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7237634Z Ran 1 test in 1.972s
2023-01-11T21:28:27.7237639Z 
2023-01-11T21:28:27.7237699Z OK
2023-01-11T21:28:27.7237703Z 
2023-01-11T21:28:27.7237773Z Generating XML reports...
2023-01-11T21:28:27.7238099Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212812.xml
2023-01-11T21:28:27.7238372Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7238493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7238769Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7238904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7239089Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9xlbnybd
2023-01-11T21:28:27.7239318Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9xlbnybd/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7239534Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7239555Z 
2023-01-11T21:28:27.7239615Z Running tests...
2023-01-11T21:28:27.7239811Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7240040Z   test_forward_backward (__main__.ReducerTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.7240337Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:28:27.7240405Z ok (0.004s)
2023-01-11T21:28:27.7240410Z 
2023-01-11T21:28:27.7240604Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7240679Z Ran 1 test in 0.011s
2023-01-11T21:28:27.7240685Z 
2023-01-11T21:28:27.7240747Z OK
2023-01-11T21:28:27.7240756Z 
2023-01-11T21:28:27.7240826Z Generating XML reports...
2023-01-11T21:28:27.7241118Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20230111212816.xml
2023-01-11T21:28:27.7241388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7241510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7241787Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7241922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7242108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlauxmvc
2023-01-11T21:28:27.7242307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlauxmvc/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7242539Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7242546Z 
2023-01-11T21:28:27.7242607Z Running tests...
2023-01-11T21:28:27.7242803Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7243040Z   test_forward_backward_optimizer (__main__.ReducerTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.7243335Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:28:27.7243929Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:28:27.7244038Z ok (0.007s)
2023-01-11T21:28:27.7244043Z 
2023-01-11T21:28:27.7244241Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7244317Z Ran 1 test in 0.011s
2023-01-11T21:28:27.7244322Z 
2023-01-11T21:28:27.7244382Z OK
2023-01-11T21:28:27.7244387Z 
2023-01-11T21:28:27.7244471Z Generating XML reports...
2023-01-11T21:28:27.7244753Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20230111212817.xml
2023-01-11T21:28:27.7245025Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7245147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7245431Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7245627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7245813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8o6qa22p
2023-01-11T21:28:27.7246010Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8o6qa22p/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7246239Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7246244Z 
2023-01-11T21:28:27.7246318Z Running tests...
2023-01-11T21:28:27.7246501Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7246750Z   test_forward_backward_unused_parameters (__main__.ReducerTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.7247047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:28:27.7247115Z ok (0.005s)
2023-01-11T21:28:27.7247124Z 
2023-01-11T21:28:27.7247315Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7247392Z Ran 1 test in 0.011s
2023-01-11T21:28:27.7247397Z 
2023-01-11T21:28:27.7247458Z OK
2023-01-11T21:28:27.7247463Z 
2023-01-11T21:28:27.7247547Z Generating XML reports...
2023-01-11T21:28:27.7247828Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20230111212819.xml
2023-01-11T21:28:27.7248101Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7248228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7248507Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7248642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7248829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9t7550by
2023-01-11T21:28:27.7249025Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9t7550by/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7249253Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7249259Z 
2023-01-11T21:28:27.7249330Z Running tests...
2023-01-11T21:28:27.7249511Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7249747Z   test_multi_dtype_multi_bucket (__main__.ReducerTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.7250047Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:28:27.7250113Z ok (0.003s)
2023-01-11T21:28:27.7250118Z 
2023-01-11T21:28:27.7250310Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7250386Z Ran 1 test in 0.011s
2023-01-11T21:28:27.7250421Z 
2023-01-11T21:28:27.7250481Z OK
2023-01-11T21:28:27.7250486Z 
2023-01-11T21:28:27.7250572Z Generating XML reports...
2023-01-11T21:28:27.7251015Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20230111212820.xml
2023-01-11T21:28:27.7251342Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7251465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7251743Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7251877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7252061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpd_gw0jmj
2023-01-11T21:28:27.7252255Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpd_gw0jmj/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7252535Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7252541Z 
2023-01-11T21:28:27.7252613Z Running tests...
2023-01-11T21:28:27.7252793Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7253028Z   test_multi_dtype_single_bucket (__main__.ReducerTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.7253325Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:28:27.7253391Z ok (0.005s)
2023-01-11T21:28:27.7253395Z 
2023-01-11T21:28:27.7253587Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7253664Z Ran 1 test in 0.011s
2023-01-11T21:28:27.7253669Z 
2023-01-11T21:28:27.7253730Z OK
2023-01-11T21:28:27.7253735Z 
2023-01-11T21:28:27.7253818Z Generating XML reports...
2023-01-11T21:28:27.7254112Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20230111212822.xml
2023-01-11T21:28:27.7254371Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7254493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7254770Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7254904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7255093Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphclv281g
2023-01-11T21:28:27.7255286Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphclv281g/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7255516Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7255521Z 
2023-01-11T21:28:27.7255598Z Running tests...
2023-01-11T21:28:27.7255791Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7256017Z   test_single_dtype_single_bucket (__main__.ReducerTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.7256314Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:28:27.7256380Z ok (0.002s)
2023-01-11T21:28:27.7256385Z 
2023-01-11T21:28:27.7256576Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7256651Z Ran 1 test in 0.011s
2023-01-11T21:28:27.7256656Z 
2023-01-11T21:28:27.7256715Z OK
2023-01-11T21:28:27.7256720Z 
2023-01-11T21:28:27.7256802Z Generating XML reports...
2023-01-11T21:28:27.7257089Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20230111212823.xml
2023-01-11T21:28:27.7257388Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7257510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7257790Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7264023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7264258Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpikx92kxu
2023-01-11T21:28:27.7264461Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpikx92kxu/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7264732Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7264739Z 
2023-01-11T21:28:27.7264813Z Running tests...
2023-01-11T21:28:27.7265018Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7265360Z   test_logging_init (__main__.RendezvousEnvTest) ... INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:28:27.7265652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T21:28:27.7265724Z ok (0.258s)
2023-01-11T21:28:27.7265731Z 
2023-01-11T21:28:27.7265925Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7266001Z Ran 1 test in 0.258s
2023-01-11T21:28:27.7266007Z 
2023-01-11T21:28:27.7266068Z OK
2023-01-11T21:28:27.7266072Z 
2023-01-11T21:28:27.7266157Z Generating XML reports...
2023-01-11T21:28:27.7266472Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20230111212825.xml
2023-01-11T21:28:27.7266749Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 56 slow tests
2023-01-11T21:28:27.7266876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:28:27.7267147Z /opt/conda/lib/python3.7/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:28:27.7267284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:28:27.7267470Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpus5k4p76
2023-01-11T21:28:27.7267665Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpus5k4p76/_remote_module_non_scriptable.py
2023-01-11T21:28:27.7267895Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_gloo
2023-01-11T21:28:27.7267901Z 
2023-01-11T21:28:27.7267974Z Running tests...
2023-01-11T21:28:27.7268170Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7268837Z   test_default_store_timeout_gloo (__main__.TimeoutTest) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/74714 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.249s)
2023-01-11T21:28:27.7268845Z 
2023-01-11T21:28:27.7269036Z ----------------------------------------------------------------------
2023-01-11T21:28:27.7269098Z Ran 1 test in 0.249s
2023-01-11T21:28:27.7269115Z 
2023-01-11T21:28:27.7269174Z OK (skipped=1)
2023-01-11T21:28:27.7269182Z 
2023-01-11T21:28:27.7269267Z Generating XML reports...
2023-01-11T21:28:27.7269562Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20230111212826.xml
2023-01-11T21:28:27.7269568Z 
2023-01-11T21:28:27.7269915Z ##[endgroup]
2023-01-11T21:28:27.7270236Z FINISHED PRINTING LOG FILE of distributed/test_c10d_gloo (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_gloo_fybruq18)
2023-01-11T21:28:27.7270242Z 
2023-01-11T21:28:27.9303298Z 
2023-01-11T21:28:27.9303868Z real	42m29.507s
2023-01-11T21:28:27.9304077Z user	86m37.949s
2023-01-11T21:28:27.9304259Z sys	22m32.661s
2023-01-11T21:28:27.9304457Z + assert_git_not_dirty
2023-01-11T21:28:27.9304855Z + [[ linux-focal-py3.7-gcc7 != *rocm* ]]
2023-01-11T21:28:27.9305138Z + [[ linux-focal-py3.7-gcc7 != *xla* ]]
2023-01-11T21:28:27.9305836Z ++ git status --porcelain
2023-01-11T21:28:32.5836826Z + git_status=
2023-01-11T21:28:32.5837248Z + [[ -n '' ]]
2023-01-11T21:28:32.5837505Z + [[ linux-focal-py3.7-gcc7 == *cuda* ]]
2023-01-11T21:28:32.5838723Z + [[ 2 == 1 ]]
2023-01-11T21:28:32.5915877Z ##[group]Run cat test/**/*.log || true
2023-01-11T21:28:32.5916165Z cat test/**/*.log || true
2023-01-11T21:28:32.6725836Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:28:32.6726068Z env:
2023-01-11T21:28:32.6726248Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:28:32.6726516Z   DOCKER_CONTAINER_ID: 16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58
2023-01-11T21:28:32.6726775Z ##[endgroup]
2023-01-11T21:28:32.6767619Z cat: test/**/*.log: No such file or directory
2023-01-11T21:28:32.6797897Z Prepare all required actions
2023-01-11T21:28:32.6798205Z Getting action download info
2023-01-11T21:28:33.1583373Z ##[group]Run ./.github/actions/get-workflow-job-id
2023-01-11T21:28:33.1583582Z with:
2023-01-11T21:28:33.1583915Z   github-token: ***
2023-01-11T21:28:33.1584083Z env:
2023-01-11T21:28:33.1584242Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:28:33.1584518Z   DOCKER_CONTAINER_ID: 16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58
2023-01-11T21:28:33.1584772Z ##[endgroup]
2023-01-11T21:28:33.1610763Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2023-01-11T21:28:33.1611129Z with:
2023-01-11T21:28:33.1611287Z   shell: bash
2023-01-11T21:28:33.1611446Z   timeout_minutes: 10
2023-01-11T21:28:33.1611624Z   max_attempts: 5
2023-01-11T21:28:33.1611805Z   retry_wait_seconds: 30
2023-01-11T21:28:33.1612205Z   command: set -eux
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}"

2023-01-11T21:28:33.1612590Z   polling_interval_seconds: 1
2023-01-11T21:28:33.1612770Z   warning_on_retry: true
2023-01-11T21:28:33.1612955Z   continue_on_error: false
2023-01-11T21:28:33.1613127Z env:
2023-01-11T21:28:33.1613282Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:28:33.1613552Z   DOCKER_CONTAINER_ID: 16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58
2023-01-11T21:28:33.1613924Z   GITHUB_TOKEN: ***
2023-01-11T21:28:33.1614095Z ##[endgroup]
2023-01-11T21:28:33.2129354Z + python3 -m pip install requests==2.26.0
2023-01-11T21:28:33.9193906Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T21:28:33.9458807Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0)
2023-01-11T21:28:33.9592502Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.14)
2023-01-11T21:28:33.9754375Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12)
2023-01-11T21:28:33.9782186Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2022.12.7)
2023-01-11T21:28:33.9799406Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.4)
2023-01-11T21:28:34.3109692Z ++ python3 .github/scripts/get_workflow_job_id.py 3896099317 i-01f7d826cf45480d0
2023-01-11T21:28:40.6763725Z + GHA_WORKFLOW_JOB_ID=10588877558
2023-01-11T21:28:40.6764395Z + echo job-id=10588877558
2023-01-11T21:28:41.2194343Z Command completed after 1 attempt(s).
2023-01-11T21:28:41.2327024Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2023-01-11T21:28:41.2327499Z kill "$MONITOR_SCRIPT_PID"
2023-01-11T21:28:41.2338961Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:28:41.2339186Z env:
2023-01-11T21:28:41.2339367Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:28:41.2339636Z   DOCKER_CONTAINER_ID: 16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58
2023-01-11T21:28:41.2339908Z   MONITOR_SCRIPT_PID: 28361
2023-01-11T21:28:41.2340094Z ##[endgroup]
2023-01-11T21:28:41.2424784Z Prepare all required actions
2023-01-11T21:28:41.2425031Z Getting action download info
2023-01-11T21:28:41.9828588Z Download action repository 'actions/upload-artifact@v3' (SHA:0b7f8abb1508181956e8e162db84b466c27e18ce)
2023-01-11T21:28:42.1361908Z ##[group]Run ./.github/actions/upload-test-artifacts
2023-01-11T21:28:42.1362118Z with:
2023-01-11T21:28:42.1362355Z   file-suffix: test-distributed-2-2-linux.2xlarge_10588877558
2023-01-11T21:28:42.1362585Z env:
2023-01-11T21:28:42.1362748Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:28:42.1363022Z   DOCKER_CONTAINER_ID: 16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58
2023-01-11T21:28:42.1363283Z ##[endgroup]
2023-01-11T21:28:42.1385059Z ##[group]Run # Remove any previous test jsons if they exist
2023-01-11T21:28:42.1385321Z # Remove any previous test jsons if they exist
2023-01-11T21:28:42.1385547Z rm -f test-jsons-*.zip
2023-01-11T21:28:42.1385820Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2023-01-11T21:28:42.1397351Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:28:42.1397570Z env:
2023-01-11T21:28:42.1397746Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:28:42.1398007Z   DOCKER_CONTAINER_ID: 16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58
2023-01-11T21:28:42.1398330Z   FILE_SUFFIX: test-distributed-2-2-linux.2xlarge_10588877558
2023-01-11T21:28:42.1398561Z ##[endgroup]
2023-01-11T21:28:42.1579804Z   adding: test/allowlist_for_publicAPI.json (deflated 78%)
2023-01-11T21:28:42.1616141Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2023-01-11T21:28:42.1632162Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2023-01-11T21:28:42.1633338Z   adding: test/.pytorch-slow-tests.json (deflated 74%)
2023-01-11T21:28:42.1637751Z   adding: test/.pytorch-disabled-tests.json (deflated 84%)
2023-01-11T21:28:42.1656433Z ##[group]Run # Remove any previous test reports if they exist
2023-01-11T21:28:42.1656712Z # Remove any previous test reports if they exist
2023-01-11T21:28:42.1656949Z rm -f test-reports-*.zip
2023-01-11T21:28:42.1657209Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv'
2023-01-11T21:28:42.1667840Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:28:42.1668055Z env:
2023-01-11T21:28:42.1668229Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:28:42.1668491Z   DOCKER_CONTAINER_ID: 16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58
2023-01-11T21:28:42.1668818Z   FILE_SUFFIX: test-distributed-2-2-linux.2xlarge_10588877558
2023-01-11T21:28:42.1669063Z ##[endgroup]
2023-01-11T21:28:42.1787478Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_ignored_modules/TEST-TestFSDPIgnoredModules-20230111204604.xml (deflated 78%)
2023-01-11T21:28:42.1799197Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_optim_state/TEST-TestFSDPOptimState-20230111204617.xml (deflated 93%)
2023-01-11T21:28:42.1802491Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-ProcessGroupShareTensorTest-20230111204830.xml (deflated 42%)
2023-01-11T21:28:42.1806606Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-ProcessGroupShareTensorTest-20230111204832.xml (deflated 42%)
2023-01-11T21:28:42.1810091Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-ProcessGroupShareTensorTest-20230111204833.xml (deflated 42%)
2023-01-11T21:28:42.1816093Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111204835.xml (deflated 43%)
2023-01-11T21:28:42.1819199Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111204837.xml (deflated 43%)
2023-01-11T21:28:42.1822657Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111204839.xml (deflated 44%)
2023-01-11T21:28:42.1825932Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111204840.xml (deflated 44%)
2023-01-11T21:28:42.1830137Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111204842.xml (deflated 44%)
2023-01-11T21:28:42.1832696Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111204844.xml (deflated 44%)
2023-01-11T21:28:42.1836329Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20230111204845.xml (deflated 61%)
2023-01-11T21:28:42.1839954Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20230111204847.xml (deflated 62%)
2023-01-11T21:28:42.1851046Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_binary_cmp/TEST-TestShardedTensorBinaryOps-20230111204848.xml (deflated 75%)
2023-01-11T21:28:42.1854691Z   adding: test/test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20230111204850.xml (deflated 82%)
2023-01-11T21:28:42.1858555Z   adding: test/test-reports/python-unittest/distributed._tools.test_memory_tracker/TEST-TestMemoryTracker-20230111204851.xml (deflated 44%)
2023-01-11T21:28:42.1864456Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_sharded_tensor_reshard/TEST-TestReshard-20230111204853.xml (deflated 63%)
2023-01-11T21:28:42.1870237Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20230111204854.xml (deflated 44%)
2023-01-11T21:28:42.1874825Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding/TEST-TestShardedEmbedding-20230111204856.xml (deflated 62%)
2023-01-11T21:28:42.1880148Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_chunk/TEST-TestShardedTensorChunkOps-20230111204857.xml (deflated 62%)
2023-01-11T21:28:42.1883529Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_optim.test_sharded_optim/TEST-TestShardedOptimizer-20230111204858.xml (deflated 61%)
2023-01-11T21:28:42.1890480Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20230111204902.xml (deflated 42%)
2023-01-11T21:28:42.1895590Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20230111204903.xml (deflated 42%)
2023-01-11T21:28:42.1899276Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20230111204905.xml (deflated 42%)
2023-01-11T21:28:42.1903183Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-ProcessGroupShareTensorTest-20230111204907.xml (deflated 42%)
2023-01-11T21:28:42.1906739Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111204908.xml (deflated 44%)
2023-01-11T21:28:42.1910308Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111204910.xml (deflated 43%)
2023-01-11T21:28:42.1913794Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111204912.xml (deflated 44%)
2023-01-11T21:28:42.1917172Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111204914.xml (deflated 43%)
2023-01-11T21:28:42.1922845Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111204915.xml (deflated 44%)
2023-01-11T21:28:42.1926326Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111204917.xml (deflated 43%)
2023-01-11T21:28:42.1926982Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111204919.xml (deflated 44%)
2023-01-11T21:28:42.1927624Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111204920.xml (deflated 44%)
2023-01-11T21:28:42.1928274Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111204922.xml (deflated 43%)
2023-01-11T21:28:42.1928903Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_init/TEST-TestShardedTensorNNInit-20230111204924.xml (deflated 71%)
2023-01-11T21:28:42.1929606Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_elementwise_ops/TEST-TestShardedTensorElementWiseOps-20230111204925.xml (deflated 76%)
2023-01-11T21:28:42.1930269Z   adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorOps-20230111204927.xml (deflated 69%)
2023-01-11T21:28:42.1930994Z   adding: test/test-reports/python-unittest/distributed._shard.test_partial_tensor/TEST-TestPartialTensorReshard-20230111204927.xml (deflated 62%)
2023-01-11T21:28:42.1931611Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_tensor_ops/TEST-TestTensorOps-20230111204928.xml (deflated 78%)
2023-01-11T21:28:42.1932235Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_linear/TEST-TestShardedTensorOpsLinear-20230111204930.xml (deflated 71%)
2023-01-11T21:28:42.1932878Z   adding: test/test-reports/python-unittest/distributed._shard.sharding_plan.test_sharding_plan/TEST-TestShardingPlan-20230111204931.xml (deflated 78%)
2023-01-11T21:28:42.1933536Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_matrix_ops/TEST-TestShardedTensorMatrixOps-20230111204932.xml (deflated 88%)
2023-01-11T21:28:42.1934160Z   adding: test/test-reports/python-unittest/distributed._composable.test_checkpoint/TEST-TestCheckpoint-20230111204934.xml (deflated 75%)
2023-01-11T21:28:42.1934711Z   adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallel-20230111204936.xml (deflated 90%)
2023-01-11T21:28:42.1935312Z   adding: test/test-reports/python-unittest/distributed.test_data_parallel/TEST-TestDataParallelDeviceTypeCPU-20230111204936.xml (deflated 91%)
2023-01-11T21:28:42.1935914Z   adding: test/test-reports/python-unittest/distributed.nn.jit.test_instantiator/TEST-TestInstantiator-20230111204937.xml (deflated 64%)
2023-01-11T21:28:42.1936499Z   adding: test/test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20230111204940.xml (deflated 43%)
2023-01-11T21:28:42.1937080Z   adding: test/test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20230111204941.xml (deflated 63%)
2023-01-11T21:28:42.1937646Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_planner/TEST-TestSavePlan-20230111204943.xml (deflated 71%)
2023-01-11T21:28:42.1938205Z   adding: test/test-reports/python-unittest/distributed._composable.test_contract/TEST-TestContract-20230111204945.xml (deflated 68%)
2023-01-11T21:28:42.1938760Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20230111204947.xml (deflated 55%)
2023-01-11T21:28:42.1939357Z   adding: test/test-reports/python-unittest/distributed._tensor.test_pointwise_ops/TEST-DistElementwiseOpsTest-20230111204948.xml (deflated 72%)
2023-01-11T21:28:42.1939988Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-StoreUtilTest-20230111204950.xml (deflated 63%)
2023-01-11T21:28:42.1940552Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.util_test/TEST-UtilTest-20230111204950.xml (deflated 69%)
2023-01-11T21:28:42.1941184Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20230111204952.xml (deflated 73%)
2023-01-11T21:28:42.1941823Z   adding: test/test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithBaseClass-20230111204954.xml (deflated 77%)
2023-01-11T21:28:42.1942441Z   adding: test/test-reports/python-unittest/distributed.test_multi_threaded_pg/TEST-TestCollectivesWithWrapper-20230111204954.xml (deflated 73%)
2023-01-11T21:28:42.1943127Z   adding: test/test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestFakeDistributedSingleProc-20230111204956.xml (deflated 59%)
2023-01-11T21:28:42.1943743Z   adding: test/test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributed-20230111204956.xml (deflated 79%)
2023-01-11T21:28:42.1944343Z   adding: test/test-reports/python-unittest/distributed.test_dynamo_distributed/TEST-TestDistributedMultiProc-20230111204956.xml (deflated 80%)
2023-01-11T21:28:42.1944918Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_traversal/TEST-TestTraversal-20230111204958.xml (deflated 42%)
2023-01-11T21:28:42.1945577Z   adding: test/test-reports/python-unittest/distributed.tensor.parallel.test_view_sharding_dim_change/TEST-TPViewShardingDimChangeTest-20230111205001.xml (deflated 44%)
2023-01-11T21:28:42.1946213Z   adding: test/test-reports/python-unittest/distributed.rpc.test_share_memory/TEST-TestRPCPickler-20230111205006.xml (deflated 42%)
2023-01-11T21:28:42.1946800Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_multiple_forward/TEST-TestMultiForward-20230111205010.xml (deflated 43%)
2023-01-11T21:28:42.1947425Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20230111205014.xml (deflated 78%)
2023-01-11T21:28:42.1948066Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_distributed_checkpoint/TEST-TestDistributedCheckpoint-20230111205019.xml (deflated 62%)
2023-01-11T21:28:42.1948719Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedFailure-20230111205024.xml (deflated 77%)
2023-01-11T21:28:42.1949371Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_checkpoint/TEST-TestDistributedCheckpointing-20230111205024.xml (deflated 61%)
2023-01-11T21:28:42.1949979Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_pure_fp16/TEST-TestPureFP16-20230111205029.xml (deflated 62%)
2023-01-11T21:28:42.1950583Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_fsdp_model_state/TEST-FsdpModelStateCheckpoint-20230111205034.xml (deflated 60%)
2023-01-11T21:28:42.1951205Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20230111205040.xml (deflated 64%)
2023-01-11T21:28:42.1951858Z   adding: test/test-reports/python-unittest/distributed.tensor.parallel.test_tp_examples/TEST-DistTensorParallelExampleTest-20230111205047.xml (deflated 66%)
2023-01-11T21:28:42.1952510Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_hybrid_shard/TEST-TestFSDPHybridShard-20230111205054.xml (deflated 66%)
2023-01-11T21:28:42.1953143Z   adding: test/test-reports/python-unittest/distributed.tensor.parallel.test_2d_parallel/TEST-Test2dParallelIntegration-20230111205102.xml (deflated 77%)
2023-01-11T21:28:42.1953739Z   adding: test/test-reports/python-unittest/distributed._tensor.test_api/TEST-DTensorAPITest-20230111205113.xml (deflated 71%)
2023-01-11T21:28:42.1954324Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20230111205124.xml (deflated 54%)
2023-01-11T21:28:42.1954973Z   adding: test/test-reports/python-unittest/distributed._tensor.test_matrix_ops/TEST-DistMatrixOpsTest-20230111205135.xml (deflated 74%)
2023-01-11T21:28:42.1955590Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_meta/TEST-TestFSDPWithMetaDevice-20230111205149.xml (deflated 86%)
2023-01-11T21:28:42.1956167Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm/TEST-TestCommunication-20230111205204.xml (deflated 91%)
2023-01-11T21:28:42.1956770Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_freezing_weights/TEST-TestFreezingWeights-20230111205220.xml (deflated 87%)
2023-01-11T21:28:42.1957338Z   adding: test/test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20230111205235.xml (deflated 79%)
2023-01-11T21:28:42.1957943Z   adding: test/test-reports/python-unittest/distributed.tensor.parallel.test_tp_style/TEST-TensorParallelStyleTest-20230111205251.xml (deflated 79%)
2023-01-11T21:28:42.1958569Z   adding: test/test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-MultiDimRedistributeTest-20230111205308.xml (deflated 42%)
2023-01-11T21:28:42.1959166Z   adding: test/test-reports/python-unittest/distributed._tensor.test_redistribute/TEST-RedistributeTest-20230111205308.xml (deflated 76%)
2023-01-11T21:28:42.1959705Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20230111205326.xml (deflated 40%)
2023-01-11T21:28:42.1960223Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20230111205328.xml (deflated 40%)
2023-01-11T21:28:42.1960727Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20230111205330.xml (deflated 40%)
2023-01-11T21:28:42.1961244Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-FileStoreTest-20230111205331.xml (deflated 40%)
2023-01-11T21:28:42.1961770Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20230111205333.xml (deflated 40%)
2023-01-11T21:28:42.1962285Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-HashStoreTest-20230111205335.xml (deflated 40%)
2023-01-11T21:28:42.1962808Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20230111205337.xml (deflated 40%)
2023-01-11T21:28:42.1963357Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixFileStoreTest-20230111205338.xml (deflated 40%)
2023-01-11T21:28:42.1963894Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixStoreTest-20230111205340.xml (deflated 40%)
2023-01-11T21:28:42.1964432Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20230111205341.xml (deflated 40%)
2023-01-11T21:28:42.1964966Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PrefixTCPStoreTest-20230111205343.xml (deflated 40%)
2023-01-11T21:28:42.1965503Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-PythonStoreTest-20230111205345.xml (deflated 40%)
2023-01-11T21:28:42.1966035Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousEnvTest-20230111205347.xml (deflated 40%)
2023-01-11T21:28:42.1966574Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20230111205348.xml (deflated 40%)
2023-01-11T21:28:42.1967096Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousFileTest-20230111205350.xml (deflated 41%)
2023-01-11T21:28:42.1967634Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20230111205352.xml (deflated 40%)
2023-01-11T21:28:42.1968168Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20230111205353.xml (deflated 40%)
2023-01-11T21:28:42.1968698Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20230111205355.xml (deflated 39%)
2023-01-11T21:28:42.1969252Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTCPTest-20230111205357.xml (deflated 40%)
2023-01-11T21:28:42.1969773Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20230111205409.xml (deflated 39%)
2023-01-11T21:28:42.1970320Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-RendezvousTest-20230111205410.xml (deflated 39%)
2023-01-11T21:28:42.1970924Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111205412.xml (deflated 39%)
2023-01-11T21:28:42.1971425Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111205414.xml (deflated 39%)
2023-01-11T21:28:42.1971936Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111205415.xml (deflated 38%)
2023-01-11T21:28:42.1972464Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111205417.xml (deflated 39%)
2023-01-11T21:28:42.1972976Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111205419.xml (deflated 38%)
2023-01-11T21:28:42.1973469Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111205421.xml (deflated 39%)
2023-01-11T21:28:42.1973980Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111205422.xml (deflated 39%)
2023-01-11T21:28:42.1974484Z   adding: test/test-reports/python-unittest/distributed.test_store/TEST-TCPStoreTest-20230111205426.xml (deflated 39%)
2023-01-11T21:28:42.1975062Z   adding: test/test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPInitialization-20230111205428.xml (deflated 75%)
2023-01-11T21:28:42.1975675Z   adding: test/test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPModelCheckpointing-20230111205428.xml (deflated 69%)
2023-01-11T21:28:42.1976299Z   adding: test/test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPOptimStateDict-20230111205428.xml (deflated 60%)
2023-01-11T21:28:42.1976886Z   adding: test/test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestFSDPRuntime-20230111205428.xml (deflated 54%)
2023-01-11T21:28:42.1977471Z   adding: test/test-reports/python-unittest/distributed._composable.test_fully_shard/TEST-TestMixedPrecision-20230111205428.xml (deflated 40%)
2023-01-11T21:28:42.1978020Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20230111205449.xml (deflated 93%)
2023-01-11T21:28:42.1978559Z   adding: test/test-reports/python-unittest/distributed._tensor.test_view_ops/TEST-TestViewOps-20230111205511.xml (deflated 53%)
2023-01-11T21:28:42.1979129Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpoint-20230111205539.xml (deflated 91%)
2023-01-11T21:28:42.1979759Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_checkpoint/TEST-TestFSDPCheckpointSubmodule-20230111205539.xml (deflated 43%)
2023-01-11T21:28:42.1980394Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsFQNs-20230111205611.xml (deflated 56%)
2023-01-11T21:28:42.1981101Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsMultipleParamGroups-20230111205611.xml (deflated 85%)
2023-01-11T21:28:42.1981807Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsNoSync-20230111205611.xml (deflated 44%)
2023-01-11T21:28:42.1982487Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsParamAccess-20230111205611.xml (deflated 45%)
2023-01-11T21:28:42.1983271Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsUnshardReshard-20230111205611.xml (deflated 78%)
2023-01-11T21:28:42.1984037Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsWriteback-20230111205611.xml (deflated 67%)
2023-01-11T21:28:42.1984716Z   adding: test/test-reports/python-unittest/distributed._tensor.test_tensor_ops/TEST-DistTensorOpsTest-20230111205643.xml (deflated 84%)
2023-01-11T21:28:42.1985272Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestAutoWrap-20230111205721.xml (deflated 87%)
2023-01-11T21:28:42.1985808Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_wrap/TEST-TestFSDPWrap-20230111205721.xml (deflated 91%)
2023-01-11T21:28:42.1986310Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205805.xml (deflated 40%)
2023-01-11T21:28:42.1986808Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205806.xml (deflated 40%)
2023-01-11T21:28:42.1987302Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205808.xml (deflated 41%)
2023-01-11T21:28:42.1987794Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205809.xml (deflated 40%)
2023-01-11T21:28:42.1988274Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205811.xml (deflated 39%)
2023-01-11T21:28:42.1988771Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205812.xml (deflated 40%)
2023-01-11T21:28:42.1989267Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205813.xml (deflated 40%)
2023-01-11T21:28:42.1989759Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205815.xml (deflated 40%)
2023-01-11T21:28:42.1990235Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205816.xml (deflated 39%)
2023-01-11T21:28:42.1990726Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205818.xml (deflated 39%)
2023-01-11T21:28:42.1991216Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205819.xml (deflated 39%)
2023-01-11T21:28:42.1991710Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205821.xml (deflated 40%)
2023-01-11T21:28:42.1992188Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205822.xml (deflated 40%)
2023-01-11T21:28:42.1992675Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205823.xml (deflated 39%)
2023-01-11T21:28:42.1993163Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205825.xml (deflated 39%)
2023-01-11T21:28:42.1993650Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205826.xml (deflated 39%)
2023-01-11T21:28:42.1994130Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205828.xml (deflated 40%)
2023-01-11T21:28:42.1994617Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205829.xml (deflated 40%)
2023-01-11T21:28:42.1995108Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CommTest-20230111205831.xml (deflated 40%)
2023-01-11T21:28:42.1995610Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205832.xml (deflated 40%)
2023-01-11T21:28:42.1996112Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205835.xml (deflated 40%)
2023-01-11T21:28:42.1996628Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205839.xml (deflated 40%)
2023-01-11T21:28:42.1997143Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205842.xml (deflated 39%)
2023-01-11T21:28:42.1997687Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205845.xml (deflated 40%)
2023-01-11T21:28:42.1998217Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205849.xml (deflated 40%)
2023-01-11T21:28:42.1998759Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205852.xml (deflated 40%)
2023-01-11T21:28:42.1999275Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205856.xml (deflated 40%)
2023-01-11T21:28:42.1999785Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205859.xml (deflated 40%)
2023-01-11T21:28:42.2000282Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-CompilerTest-20230111205903.xml (deflated 40%)
2023-01-11T21:28:42.2000848Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205906.xml (deflated 42%)
2023-01-11T21:28:42.2001462Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205907.xml (deflated 42%)
2023-01-11T21:28:42.2002075Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205909.xml (deflated 42%)
2023-01-11T21:28:42.2002666Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205910.xml (deflated 42%)
2023-01-11T21:28:42.2003271Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205912.xml (deflated 42%)
2023-01-11T21:28:42.2003867Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205913.xml (deflated 42%)
2023-01-11T21:28:42.2004476Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205914.xml (deflated 42%)
2023-01-11T21:28:42.2005063Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205916.xml (deflated 42%)
2023-01-11T21:28:42.2005665Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205917.xml (deflated 43%)
2023-01-11T21:28:42.2006261Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205919.xml (deflated 44%)
2023-01-11T21:28:42.2006855Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205922.xml (deflated 45%)
2023-01-11T21:28:42.2007451Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205925.xml (deflated 43%)
2023-01-11T21:28:42.2008035Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205929.xml (deflated 43%)
2023-01-11T21:28:42.2008632Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205932.xml (deflated 45%)
2023-01-11T21:28:42.2009228Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205935.xml (deflated 45%)
2023-01-11T21:28:42.2009824Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205939.xml (deflated 46%)
2023-01-11T21:28:42.2010400Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205942.xml (deflated 46%)
2023-01-11T21:28:42.2011082Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205945.xml (deflated 44%)
2023-01-11T21:28:42.2011686Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205949.xml (deflated 45%)
2023-01-11T21:28:42.2012337Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205952.xml (deflated 45%)
2023-01-11T21:28:42.2012923Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205955.xml (deflated 44%)
2023-01-11T21:28:42.2013559Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111205959.xml (deflated 44%)
2023-01-11T21:28:42.2014158Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210002.xml (deflated 43%)
2023-01-11T21:28:42.2014759Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210004.xml (deflated 42%)
2023-01-11T21:28:42.2015340Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210005.xml (deflated 42%)
2023-01-11T21:28:42.2015933Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210007.xml (deflated 45%)
2023-01-11T21:28:42.2016523Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210008.xml (deflated 45%)
2023-01-11T21:28:42.2017122Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210009.xml (deflated 42%)
2023-01-11T21:28:42.2017703Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210011.xml (deflated 45%)
2023-01-11T21:28:42.2018326Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210012.xml (deflated 43%)
2023-01-11T21:28:42.2018922Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210014.xml (deflated 42%)
2023-01-11T21:28:42.2019516Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210015.xml (deflated 43%)
2023-01-11T21:28:42.2020100Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210017.xml (deflated 42%)
2023-01-11T21:28:42.2020684Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210018.xml (deflated 43%)
2023-01-11T21:28:42.2021276Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210019.xml (deflated 42%)
2023-01-11T21:28:42.2021867Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210021.xml (deflated 42%)
2023-01-11T21:28:42.2022468Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210022.xml (deflated 42%)
2023-01-11T21:28:42.2023124Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210024.xml (deflated 42%)
2023-01-11T21:28:42.2023723Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210025.xml (deflated 41%)
2023-01-11T21:28:42.2024324Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210027.xml (deflated 42%)
2023-01-11T21:28:42.2024872Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-TimeoutTest-20230111210201.xml (deflated 41%)
2023-01-11T21:28:42.2025420Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210028.xml (deflated 43%)
2023-01-11T21:28:42.2026020Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210030.xml (deflated 43%)
2023-01-11T21:28:42.2026653Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210031.xml (deflated 43%)
2023-01-11T21:28:42.2027252Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210032.xml (deflated 43%)
2023-01-11T21:28:42.2027859Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210034.xml (deflated 42%)
2023-01-11T21:28:42.2028452Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210035.xml (deflated 42%)
2023-01-11T21:28:42.2029042Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210037.xml (deflated 42%)
2023-01-11T21:28:42.2029639Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210040.xml (deflated 43%)
2023-01-11T21:28:42.2030218Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210042.xml (deflated 43%)
2023-01-11T21:28:42.2030816Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210043.xml (deflated 42%)
2023-01-11T21:28:42.2031411Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210044.xml (deflated 43%)
2023-01-11T21:28:42.2032003Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210046.xml (deflated 43%)
2023-01-11T21:28:42.2032584Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210047.xml (deflated 43%)
2023-01-11T21:28:42.2033175Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210049.xml (deflated 43%)
2023-01-11T21:28:42.2033764Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210050.xml (deflated 43%)
2023-01-11T21:28:42.2034364Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210052.xml (deflated 44%)
2023-01-11T21:28:42.2034944Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210053.xml (deflated 44%)
2023-01-11T21:28:42.2035539Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210055.xml (deflated 43%)
2023-01-11T21:28:42.2036126Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210056.xml (deflated 45%)
2023-01-11T21:28:42.2036717Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210058.xml (deflated 42%)
2023-01-11T21:28:42.2037300Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210059.xml (deflated 43%)
2023-01-11T21:28:42.2037894Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210100.xml (deflated 42%)
2023-01-11T21:28:42.2038487Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210102.xml (deflated 42%)
2023-01-11T21:28:42.2039077Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210103.xml (deflated 42%)
2023-01-11T21:28:42.2039672Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-DistributedDataParallelTest-20230111210107.xml (deflated 42%)
2023-01-11T21:28:42.2040241Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111210110.xml (deflated 42%)
2023-01-11T21:28:42.2040809Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111210112.xml (deflated 42%)
2023-01-11T21:28:42.2041413Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111210113.xml (deflated 42%)
2023-01-11T21:28:42.2041980Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111210114.xml (deflated 42%)
2023-01-11T21:28:42.2042559Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111210116.xml (deflated 42%)
2023-01-11T21:28:42.2043124Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111210117.xml (deflated 42%)
2023-01-11T21:28:42.2043689Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111210119.xml (deflated 42%)
2023-01-11T21:28:42.2044251Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111210120.xml (deflated 43%)
2023-01-11T21:28:42.2044799Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclErrorHandlingTest-20230111210122.xml (deflated 42%)
2023-01-11T21:28:42.2045469Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111210123.xml (deflated 44%)
2023-01-11T21:28:42.2046217Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111210124.xml (deflated 44%)
2023-01-11T21:28:42.2046956Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111210126.xml (deflated 44%)
2023-01-11T21:28:42.2047677Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111210127.xml (deflated 44%)
2023-01-11T21:28:42.2048413Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-NcclProcessGroupWithDispatchedCollectivesTests-20230111210129.xml (deflated 44%)
2023-01-11T21:28:42.2049086Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLNoGPUTest-20230111210130.xml (deflated 42%)
2023-01-11T21:28:42.2049703Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210132.xml (deflated 42%)
2023-01-11T21:28:42.2050274Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210133.xml (deflated 42%)
2023-01-11T21:28:42.2051420Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210135.xml (deflated 42%)
2023-01-11T21:28:42.2051992Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210136.xml (deflated 42%)
2023-01-11T21:28:42.2052559Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210137.xml (deflated 42%)
2023-01-11T21:28:42.2053124Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210139.xml (deflated 42%)
2023-01-11T21:28:42.2053671Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210140.xml (deflated 42%)
2023-01-11T21:28:42.2054236Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210142.xml (deflated 42%)
2023-01-11T21:28:42.2054804Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210143.xml (deflated 42%)
2023-01-11T21:28:42.2055364Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210145.xml (deflated 42%)
2023-01-11T21:28:42.2055915Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210146.xml (deflated 43%)
2023-01-11T21:28:42.2056473Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210148.xml (deflated 42%)
2023-01-11T21:28:42.2057092Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210149.xml (deflated 41%)
2023-01-11T21:28:42.2057653Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210150.xml (deflated 42%)
2023-01-11T21:28:42.2058235Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210152.xml (deflated 42%)
2023-01-11T21:28:42.2058800Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210153.xml (deflated 42%)
2023-01-11T21:28:42.2059357Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210155.xml (deflated 42%)
2023-01-11T21:28:42.2059916Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210156.xml (deflated 42%)
2023-01-11T21:28:42.2060462Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-ProcessGroupNCCLTest-20230111210158.xml (deflated 42%)
2023-01-11T21:28:42.2061011Z   adding: test/test-reports/python-unittest/distributed.test_c10d_nccl/TEST-RendezvousEnvTest-20230111210159.xml (deflated 41%)
2023-01-11T21:28:42.2061606Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParams-20230111210202.xml (deflated 93%)
2023-01-11T21:28:42.2062251Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_summon_full_params/TEST-TestSummonFullParamsNoShard-20230111210202.xml (deflated 43%)
2023-01-11T21:28:42.2062822Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestHooks-20230111210328.xml (deflated 81%)
2023-01-11T21:28:42.2063416Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestNoGrad-20230111210328.xml (deflated 64%)
2023-01-11T21:28:42.2063964Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParamInit-20230111210328.xml (deflated 62%)
2023-01-11T21:28:42.2064529Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_core/TEST-TestParityWithDDP-20230111210328.xml (deflated 93%)
2023-01-11T21:28:42.2065176Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPDifferentSubmodulePrecision-20230111210505.xml (deflated 80%)
2023-01-11T21:28:42.2065901Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionIgnoredModules-20230111210505.xml (deflated 44%)
2023-01-11T21:28:42.2066602Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20230111210505.xml (deflated 93%)
2023-01-11T21:28:42.2067293Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20230111210505.xml (deflated 68%)
2023-01-11T21:28:42.2067910Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_state_dict/TEST-TestFSDPStateDict-20230111210644.xml (deflated 96%)
2023-01-11T21:28:42.2068547Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20230111210955.xml (deflated 42%)
2023-01-11T21:28:42.2069224Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentDistAutogradTest-20230111211007.xml (deflated 42%)
2023-01-11T21:28:42.2069865Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211017.xml (deflated 41%)
2023-01-11T21:28:42.2070472Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211027.xml (deflated 41%)
2023-01-11T21:28:42.2071063Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211037.xml (deflated 42%)
2023-01-11T21:28:42.2071708Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211052.xml (deflated 42%)
2023-01-11T21:28:42.2072310Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211103.xml (deflated 42%)
2023-01-11T21:28:42.2072943Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211107.xml (deflated 43%)
2023-01-11T21:28:42.2073529Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211117.xml (deflated 41%)
2023-01-11T21:28:42.2074129Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211126.xml (deflated 41%)
2023-01-11T21:28:42.2074741Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211133.xml (deflated 42%)
2023-01-11T21:28:42.2075349Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211143.xml (deflated 41%)
2023-01-11T21:28:42.2075938Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211153.xml (deflated 41%)
2023-01-11T21:28:42.2076542Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211202.xml (deflated 42%)
2023-01-11T21:28:42.2077149Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211208.xml (deflated 42%)
2023-01-11T21:28:42.2077751Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211226.xml (deflated 41%)
2023-01-11T21:28:42.2078335Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211238.xml (deflated 41%)
2023-01-11T21:28:42.2078935Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211244.xml (deflated 41%)
2023-01-11T21:28:42.2079535Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211254.xml (deflated 41%)
2023-01-11T21:28:42.2080145Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211301.xml (deflated 42%)
2023-01-11T21:28:42.2080737Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211312.xml (deflated 41%)
2023-01-11T21:28:42.2081337Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyFaultyAgentRpcTest-20230111211322.xml (deflated 41%)
2023-01-11T21:28:42.2081957Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20230111211332.xml (deflated 42%)
2023-01-11T21:28:42.2082590Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20230111211342.xml (deflated 41%)
2023-01-11T21:28:42.2083209Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20230111211352.xml (deflated 42%)
2023-01-11T21:28:42.2083841Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20230111211402.xml (deflated 42%)
2023-01-11T21:28:42.2084462Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20230111211409.xml (deflated 42%)
2023-01-11T21:28:42.2085093Z   adding: test/test-reports/python-unittest/distributed.rpc.test_faulty_agent/TEST-FaultyJitFaultyAgentRpcTest-20230111211418.xml (deflated 42%)
2023-01-11T21:28:42.2085747Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20230111211428.xml (deflated 42%)
2023-01-11T21:28:42.2086528Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20230111211430.xml (deflated 43%)
2023-01-11T21:28:42.2087250Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20230111211434.xml (deflated 42%)
2023-01-11T21:28:42.2087991Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20230111211438.xml (deflated 42%)
2023-01-11T21:28:42.2088676Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111211442.xml (deflated 43%)
2023-01-11T21:28:42.2089356Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111211446.xml (deflated 42%)
2023-01-11T21:28:42.2090019Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111211450.xml (deflated 43%)
2023-01-11T21:28:42.2090706Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111211454.xml (deflated 43%)
2023-01-11T21:28:42.2091454Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20230111211458.xml (deflated 42%)
2023-01-11T21:28:42.2092101Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111211502.xml (deflated 41%)
2023-01-11T21:28:42.2092765Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111211506.xml (deflated 41%)
2023-01-11T21:28:42.2093411Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111211510.xml (deflated 42%)
2023-01-11T21:28:42.2094071Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111211514.xml (deflated 41%)
2023-01-11T21:28:42.2094731Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111211518.xml (deflated 41%)
2023-01-11T21:28:42.2095384Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111211520.xml (deflated 41%)
2023-01-11T21:28:42.2096022Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111211521.xml (deflated 41%)
2023-01-11T21:28:42.2096676Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111211523.xml (deflated 41%)
2023-01-11T21:28:42.2097377Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211525.xml (deflated 43%)
2023-01-11T21:28:42.2098112Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211529.xml (deflated 44%)
2023-01-11T21:28:42.2098833Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211533.xml (deflated 44%)
2023-01-11T21:28:42.2099565Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211537.xml (deflated 45%)
2023-01-11T21:28:42.2100295Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211541.xml (deflated 44%)
2023-01-11T21:28:42.2101024Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211545.xml (deflated 44%)
2023-01-11T21:28:42.2101808Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211549.xml (deflated 44%)
2023-01-11T21:28:42.2102565Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211554.xml (deflated 44%)
2023-01-11T21:28:42.2103362Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211558.xml (deflated 44%)
2023-01-11T21:28:42.2104091Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211602.xml (deflated 45%)
2023-01-11T21:28:42.2104820Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211606.xml (deflated 44%)
2023-01-11T21:28:42.2105541Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211610.xml (deflated 44%)
2023-01-11T21:28:42.2106270Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211614.xml (deflated 45%)
2023-01-11T21:28:42.2107008Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211618.xml (deflated 44%)
2023-01-11T21:28:42.2107737Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211622.xml (deflated 44%)
2023-01-11T21:28:42.2108464Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211627.xml (deflated 44%)
2023-01-11T21:28:42.2109183Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211631.xml (deflated 43%)
2023-01-11T21:28:42.2109909Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211633.xml (deflated 44%)
2023-01-11T21:28:42.2110634Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211637.xml (deflated 44%)
2023-01-11T21:28:42.2111358Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211641.xml (deflated 44%)
2023-01-11T21:28:42.2112069Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211645.xml (deflated 43%)
2023-01-11T21:28:42.2112788Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211649.xml (deflated 44%)
2023-01-11T21:28:42.2113517Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211653.xml (deflated 44%)
2023-01-11T21:28:42.2114239Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211657.xml (deflated 44%)
2023-01-11T21:28:42.2114960Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211701.xml (deflated 43%)
2023-01-11T21:28:42.2115668Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211704.xml (deflated 44%)
2023-01-11T21:28:42.2116393Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211708.xml (deflated 44%)
2023-01-11T21:28:42.2117149Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211712.xml (deflated 44%)
2023-01-11T21:28:42.2117877Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211716.xml (deflated 44%)
2023-01-11T21:28:42.2118622Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211720.xml (deflated 44%)
2023-01-11T21:28:42.2119346Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211724.xml (deflated 44%)
2023-01-11T21:28:42.2120068Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211728.xml (deflated 44%)
2023-01-11T21:28:42.2120797Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211733.xml (deflated 44%)
2023-01-11T21:28:42.2121503Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211737.xml (deflated 44%)
2023-01-11T21:28:42.2122228Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211741.xml (deflated 44%)
2023-01-11T21:28:42.2122948Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211745.xml (deflated 44%)
2023-01-11T21:28:42.2123680Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211749.xml (deflated 44%)
2023-01-11T21:28:42.2124410Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211753.xml (deflated 44%)
2023-01-11T21:28:42.2125125Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211757.xml (deflated 44%)
2023-01-11T21:28:42.2125850Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211801.xml (deflated 43%)
2023-01-11T21:28:42.2126573Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211805.xml (deflated 44%)
2023-01-11T21:28:42.2127295Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211810.xml (deflated 44%)
2023-01-11T21:28:42.2128007Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211814.xml (deflated 44%)
2023-01-11T21:28:42.2128727Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211818.xml (deflated 44%)
2023-01-11T21:28:42.2129444Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211822.xml (deflated 44%)
2023-01-11T21:28:42.2130170Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211826.xml (deflated 44%)
2023-01-11T21:28:42.2130970Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211830.xml (deflated 44%)
2023-01-11T21:28:42.2131683Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211834.xml (deflated 45%)
2023-01-11T21:28:42.2132460Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211838.xml (deflated 45%)
2023-01-11T21:28:42.2133190Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211843.xml (deflated 44%)
2023-01-11T21:28:42.2133951Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211847.xml (deflated 44%)
2023-01-11T21:28:42.2134666Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211851.xml (deflated 44%)
2023-01-11T21:28:42.2135394Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211855.xml (deflated 44%)
2023-01-11T21:28:42.2136120Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211859.xml (deflated 44%)
2023-01-11T21:28:42.2136847Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211903.xml (deflated 44%)
2023-01-11T21:28:42.2137576Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211907.xml (deflated 44%)
2023-01-11T21:28:42.2138289Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211911.xml (deflated 44%)
2023-01-11T21:28:42.2139015Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211915.xml (deflated 44%)
2023-01-11T21:28:42.2139738Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211919.xml (deflated 44%)
2023-01-11T21:28:42.2140459Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211923.xml (deflated 44%)
2023-01-11T21:28:42.2141586Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211927.xml (deflated 44%)
2023-01-11T21:28:42.2142690Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211932.xml (deflated 44%)
2023-01-11T21:28:42.2143492Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211936.xml (deflated 43%)
2023-01-11T21:28:42.2144222Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211940.xml (deflated 44%)
2023-01-11T21:28:42.2144949Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211944.xml (deflated 44%)
2023-01-11T21:28:42.2145663Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211948.xml (deflated 44%)
2023-01-11T21:28:42.2146394Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211952.xml (deflated 44%)
2023-01-11T21:28:42.2147122Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111211956.xml (deflated 44%)
2023-01-11T21:28:42.2147848Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212000.xml (deflated 43%)
2023-01-11T21:28:42.2148556Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212004.xml (deflated 43%)
2023-01-11T21:28:42.2149334Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212008.xml (deflated 43%)
2023-01-11T21:28:42.2150107Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212013.xml (deflated 43%)
2023-01-11T21:28:42.2150839Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212017.xml (deflated 44%)
2023-01-11T21:28:42.2151564Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212021.xml (deflated 44%)
2023-01-11T21:28:42.2152269Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212025.xml (deflated 43%)
2023-01-11T21:28:42.2152992Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212027.xml (deflated 43%)
2023-01-11T21:28:42.2153712Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212031.xml (deflated 43%)
2023-01-11T21:28:42.2154436Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212035.xml (deflated 43%)
2023-01-11T21:28:42.2155143Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212039.xml (deflated 43%)
2023-01-11T21:28:42.2155868Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212043.xml (deflated 43%)
2023-01-11T21:28:42.2156594Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212047.xml (deflated 43%)
2023-01-11T21:28:42.2157315Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212052.xml (deflated 44%)
2023-01-11T21:28:42.2158026Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212056.xml (deflated 43%)
2023-01-11T21:28:42.2158748Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212100.xml (deflated 44%)
2023-01-11T21:28:42.2159550Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212104.xml (deflated 43%)
2023-01-11T21:28:42.2160261Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212108.xml (deflated 44%)
2023-01-11T21:28:42.2160992Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111212112.xml (deflated 44%)
2023-01-11T21:28:42.2161741Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20230111212116.xml (deflated 45%)
2023-01-11T21:28:42.2162507Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20230111212121.xml (deflated 45%)
2023-01-11T21:28:42.2163254Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20230111212125.xml (deflated 44%)
2023-01-11T21:28:42.2163873Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212130.xml (deflated 40%)
2023-01-11T21:28:42.2164402Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212133.xml (deflated 41%)
2023-01-11T21:28:42.2164902Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212137.xml (deflated 39%)
2023-01-11T21:28:42.2165413Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212140.xml (deflated 40%)
2023-01-11T21:28:42.2165911Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212144.xml (deflated 40%)
2023-01-11T21:28:42.2166400Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212147.xml (deflated 40%)
2023-01-11T21:28:42.2166890Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212151.xml (deflated 40%)
2023-01-11T21:28:42.2167372Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212154.xml (deflated 40%)
2023-01-11T21:28:42.2167865Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212157.xml (deflated 40%)
2023-01-11T21:28:42.2168356Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212201.xml (deflated 40%)
2023-01-11T21:28:42.2168852Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CommTest-20230111212204.xml (deflated 40%)
2023-01-11T21:28:42.2169346Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212208.xml (deflated 39%)
2023-01-11T21:28:42.2169862Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212211.xml (deflated 40%)
2023-01-11T21:28:42.2170372Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212214.xml (deflated 39%)
2023-01-11T21:28:42.2170971Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212218.xml (deflated 40%)
2023-01-11T21:28:42.2171471Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212221.xml (deflated 39%)
2023-01-11T21:28:42.2171990Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212225.xml (deflated 40%)
2023-01-11T21:28:42.2172505Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212228.xml (deflated 39%)
2023-01-11T21:28:42.2173016Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212232.xml (deflated 40%)
2023-01-11T21:28:42.2173510Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212235.xml (deflated 39%)
2023-01-11T21:28:42.2174028Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212239.xml (deflated 39%)
2023-01-11T21:28:42.2174537Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-CompilerTest-20230111212242.xml (deflated 40%)
2023-01-11T21:28:42.2175107Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212246.xml (deflated 44%)
2023-01-11T21:28:42.2175700Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212249.xml (deflated 45%)
2023-01-11T21:28:42.2176303Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212253.xml (deflated 43%)
2023-01-11T21:28:42.2176907Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212256.xml (deflated 43%)
2023-01-11T21:28:42.2177516Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212259.xml (deflated 45%)
2023-01-11T21:28:42.2178109Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212303.xml (deflated 45%)
2023-01-11T21:28:42.2178765Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212306.xml (deflated 46%)
2023-01-11T21:28:42.2179368Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212310.xml (deflated 47%)
2023-01-11T21:28:42.2180008Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212313.xml (deflated 45%)
2023-01-11T21:28:42.2180606Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212316.xml (deflated 45%)
2023-01-11T21:28:42.2181205Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212320.xml (deflated 45%)
2023-01-11T21:28:42.2181809Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212323.xml (deflated 44%)
2023-01-11T21:28:42.2182413Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212326.xml (deflated 44%)
2023-01-11T21:28:42.2183052Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212330.xml (deflated 44%)
2023-01-11T21:28:42.2183659Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212333.xml (deflated 44%)
2023-01-11T21:28:42.2184254Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212337.xml (deflated 45%)
2023-01-11T21:28:42.2184860Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212340.xml (deflated 45%)
2023-01-11T21:28:42.2185444Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212344.xml (deflated 45%)
2023-01-11T21:28:42.2186050Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212347.xml (deflated 45%)
2023-01-11T21:28:42.2186646Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212351.xml (deflated 50%)
2023-01-11T21:28:42.2187247Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212354.xml (deflated 42%)
2023-01-11T21:28:42.2187835Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212357.xml (deflated 42%)
2023-01-11T21:28:42.2188431Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212401.xml (deflated 42%)
2023-01-11T21:28:42.2189023Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212404.xml (deflated 42%)
2023-01-11T21:28:42.2189621Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212407.xml (deflated 43%)
2023-01-11T21:28:42.2190200Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212411.xml (deflated 42%)
2023-01-11T21:28:42.2190799Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212414.xml (deflated 42%)
2023-01-11T21:28:42.2191390Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212417.xml (deflated 42%)
2023-01-11T21:28:42.2191982Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212421.xml (deflated 42%)
2023-01-11T21:28:42.2192578Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212424.xml (deflated 45%)
2023-01-11T21:28:42.2193213Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212428.xml (deflated 46%)
2023-01-11T21:28:42.2193802Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212431.xml (deflated 42%)
2023-01-11T21:28:42.2194426Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212435.xml (deflated 41%)
2023-01-11T21:28:42.2195018Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212438.xml (deflated 42%)
2023-01-11T21:28:42.2195602Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212442.xml (deflated 42%)
2023-01-11T21:28:42.2196194Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212445.xml (deflated 42%)
2023-01-11T21:28:42.2196785Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212448.xml (deflated 42%)
2023-01-11T21:28:42.2197388Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-DistributedDataParallelTest-20230111212452.xml (deflated 42%)
2023-01-11T21:28:42.2198063Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20230111212455.xml (deflated 43%)
2023-01-11T21:28:42.2198808Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20230111212459.xml (deflated 43%)
2023-01-11T21:28:42.2199547Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20230111212502.xml (deflated 43%)
2023-01-11T21:28:42.2200279Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20230111212505.xml (deflated 44%)
2023-01-11T21:28:42.2201001Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-GlooProcessGroupWithDispatchedCollectivesTests-20230111212509.xml (deflated 43%)
2023-01-11T21:28:42.2201644Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212512.xml (deflated 40%)
2023-01-11T21:28:42.2202218Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212515.xml (deflated 41%)
2023-01-11T21:28:42.2202789Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212519.xml (deflated 40%)
2023-01-11T21:28:42.2203343Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212523.xml (deflated 40%)
2023-01-11T21:28:42.2203901Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212526.xml (deflated 40%)
2023-01-11T21:28:42.2204467Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212530.xml (deflated 40%)
2023-01-11T21:28:42.2205030Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212534.xml (deflated 40%)
2023-01-11T21:28:42.2205578Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212539.xml (deflated 41%)
2023-01-11T21:28:42.2206138Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212542.xml (deflated 40%)
2023-01-11T21:28:42.2206696Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212546.xml (deflated 41%)
2023-01-11T21:28:42.2207260Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212549.xml (deflated 40%)
2023-01-11T21:28:42.2207804Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212553.xml (deflated 40%)
2023-01-11T21:28:42.2208393Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212557.xml (deflated 40%)
2023-01-11T21:28:42.2208953Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212600.xml (deflated 40%)
2023-01-11T21:28:42.2209541Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212604.xml (deflated 40%)
2023-01-11T21:28:42.2210088Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212607.xml (deflated 40%)
2023-01-11T21:28:42.2210645Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212611.xml (deflated 41%)
2023-01-11T21:28:42.2211296Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212614.xml (deflated 40%)
2023-01-11T21:28:42.2211864Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212619.xml (deflated 40%)
2023-01-11T21:28:42.2212498Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212623.xml (deflated 41%)
2023-01-11T21:28:42.2213061Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212626.xml (deflated 40%)
2023-01-11T21:28:42.2213627Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212630.xml (deflated 41%)
2023-01-11T21:28:42.2214190Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212634.xml (deflated 41%)
2023-01-11T21:28:42.2214735Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212637.xml (deflated 40%)
2023-01-11T21:28:42.2215295Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212641.xml (deflated 40%)
2023-01-11T21:28:42.2215860Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212645.xml (deflated 41%)
2023-01-11T21:28:42.2216424Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212648.xml (deflated 40%)
2023-01-11T21:28:42.2216971Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212652.xml (deflated 40%)
2023-01-11T21:28:42.2217531Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212656.xml (deflated 41%)
2023-01-11T21:28:42.2218091Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212659.xml (deflated 40%)
2023-01-11T21:28:42.2218647Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212703.xml (deflated 40%)
2023-01-11T21:28:42.2219193Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212707.xml (deflated 40%)
2023-01-11T21:28:42.2219749Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212712.xml (deflated 41%)
2023-01-11T21:28:42.2220308Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212715.xml (deflated 40%)
2023-01-11T21:28:42.2220866Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212719.xml (deflated 40%)
2023-01-11T21:28:42.2221410Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212723.xml (deflated 41%)
2023-01-11T21:28:42.2221967Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212726.xml (deflated 40%)
2023-01-11T21:28:42.2222580Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212730.xml (deflated 40%)
2023-01-11T21:28:42.2223208Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212735.xml (deflated 41%)
2023-01-11T21:28:42.2223793Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212738.xml (deflated 40%)
2023-01-11T21:28:42.2224357Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212742.xml (deflated 40%)
2023-01-11T21:28:42.2224919Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212746.xml (deflated 40%)
2023-01-11T21:28:42.2225485Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212750.xml (deflated 41%)
2023-01-11T21:28:42.2226027Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212753.xml (deflated 40%)
2023-01-11T21:28:42.2226588Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212757.xml (deflated 40%)
2023-01-11T21:28:42.2227147Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212802.xml (deflated 42%)
2023-01-11T21:28:42.2227705Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212804.xml (deflated 40%)
2023-01-11T21:28:42.2228247Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212807.xml (deflated 42%)
2023-01-11T21:28:42.2228804Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212809.xml (deflated 42%)
2023-01-11T21:28:42.2229359Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ProcessGroupGlooTest-20230111212812.xml (deflated 40%)
2023-01-11T21:28:42.2229893Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20230111212816.xml (deflated 39%)
2023-01-11T21:28:42.2230395Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20230111212817.xml (deflated 39%)
2023-01-11T21:28:42.2230907Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20230111212819.xml (deflated 39%)
2023-01-11T21:28:42.2231410Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20230111212820.xml (deflated 40%)
2023-01-11T21:28:42.2231917Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20230111212822.xml (deflated 39%)
2023-01-11T21:28:42.2232409Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-ReducerTest-20230111212823.xml (deflated 40%)
2023-01-11T21:28:42.2232939Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-RendezvousEnvTest-20230111212825.xml (deflated 40%)
2023-01-11T21:28:42.2233466Z   adding: test/test-reports/python-unittest/distributed.test_c10d_gloo/TEST-TimeoutTest-20230111212826.xml (deflated 41%)
2023-01-11T21:28:42.2250008Z ##[group]Run # Remove any previous test reports if they exist
2023-01-11T21:28:42.2250279Z # Remove any previous test reports if they exist
2023-01-11T21:28:42.2250508Z rm -f usage-log-*.zip
2023-01-11T21:28:42.2250783Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2023-01-11T21:28:42.2251187Z # so check to see if the file exists first
2023-01-11T21:28:42.2251396Z if [ -f 'usage_log.txt' ]; then
2023-01-11T21:28:42.2251636Z     zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt'
2023-01-11T21:28:42.2251847Z fi
2023-01-11T21:28:42.2262072Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:28:42.2262285Z env:
2023-01-11T21:28:42.2262460Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:28:42.2262796Z   DOCKER_CONTAINER_ID: 16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58
2023-01-11T21:28:42.2263208Z   FILE_SUFFIX: test-distributed-2-2-linux.2xlarge_10588877558
2023-01-11T21:28:42.2263439Z ##[endgroup]
2023-01-11T21:28:42.2583441Z   adding: usage_log.txt (deflated 95%)
2023-01-11T21:28:42.2618208Z ##[group]Run seemethere/upload-artifact-s3@v5
2023-01-11T21:28:42.2618416Z with:
2023-01-11T21:28:42.2618700Z   s3-prefix: pytorch/pytorch/3896099317/2/artifact

2023-01-11T21:28:42.2618903Z   retention-days: 14
2023-01-11T21:28:42.2619094Z   if-no-files-found: warn
2023-01-11T21:28:42.2619291Z   path: test-jsons-*.zip
2023-01-11T21:28:42.2619459Z   name: artifact
2023-01-11T21:28:42.2619638Z   s3-bucket: gha-artifacts
2023-01-11T21:28:42.2619829Z   region: us-east-1
2023-01-11T21:28:42.2619981Z env:
2023-01-11T21:28:42.2620154Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:28:42.2620429Z   DOCKER_CONTAINER_ID: 16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58
2023-01-11T21:28:42.2620685Z ##[endgroup]
2023-01-11T21:28:42.6267052Z NOTE: s3-prefix specified, ignoring name parameter
2023-01-11T21:28:42.6267704Z With the provided path, there will be 1 file uploaded
2023-01-11T21:28:42.6268002Z Uploading to s3 prefix: pytorch/pytorch/3896099317/2/artifact
2023-01-11T21:28:42.6274426Z Starting upload of test-jsons-test-distributed-2-2-linux.2xlarge_10588877558.zip
2023-01-11T21:28:42.7148790Z Finished upload of test-jsons-test-distributed-2-2-linux.2xlarge_10588877558.zip
2023-01-11T21:28:42.7285582Z ##[group]Run seemethere/upload-artifact-s3@v5
2023-01-11T21:28:42.7285792Z with:
2023-01-11T21:28:42.7285992Z   s3-prefix: pytorch/pytorch/3896099317/2/artifact

2023-01-11T21:28:42.7286194Z   retention-days: 14
2023-01-11T21:28:42.7286388Z   if-no-files-found: error
2023-01-11T21:28:42.7286590Z   path: test-reports-*.zip
2023-01-11T21:28:42.7286761Z   name: artifact
2023-01-11T21:28:42.7286937Z   s3-bucket: gha-artifacts
2023-01-11T21:28:42.7287122Z   region: us-east-1
2023-01-11T21:28:42.7287278Z env:
2023-01-11T21:28:42.7287446Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:28:42.7287715Z   DOCKER_CONTAINER_ID: 16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58
2023-01-11T21:28:42.7287956Z ##[endgroup]
2023-01-11T21:28:43.0629146Z NOTE: s3-prefix specified, ignoring name parameter
2023-01-11T21:28:43.0629678Z With the provided path, there will be 1 file uploaded
2023-01-11T21:28:43.0629969Z Uploading to s3 prefix: pytorch/pytorch/3896099317/2/artifact
2023-01-11T21:28:43.0636707Z Starting upload of test-reports-test-distributed-2-2-linux.2xlarge_10588877558.zip
2023-01-11T21:28:43.2214266Z Finished upload of test-reports-test-distributed-2-2-linux.2xlarge_10588877558.zip
2023-01-11T21:28:43.2370610Z ##[group]Run seemethere/upload-artifact-s3@v5
2023-01-11T21:28:43.2370811Z with:
2023-01-11T21:28:43.2371187Z   s3-prefix: pytorch/pytorch/3896099317/2/artifact

2023-01-11T21:28:43.2371404Z   retention-days: 14
2023-01-11T21:28:43.2371597Z   if-no-files-found: ignore
2023-01-11T21:28:43.2371788Z   path: usage-log-*.zip
2023-01-11T21:28:43.2371967Z   name: artifact
2023-01-11T21:28:43.2372144Z   s3-bucket: gha-artifacts
2023-01-11T21:28:43.2372319Z   region: us-east-1
2023-01-11T21:28:43.2372481Z env:
2023-01-11T21:28:43.2372646Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:28:43.2372898Z   DOCKER_CONTAINER_ID: 16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58
2023-01-11T21:28:43.2373152Z ##[endgroup]
2023-01-11T21:28:43.5700345Z NOTE: s3-prefix specified, ignoring name parameter
2023-01-11T21:28:43.5700866Z With the provided path, there will be 1 file uploaded
2023-01-11T21:28:43.5701321Z Uploading to s3 prefix: pytorch/pytorch/3896099317/2/artifact
2023-01-11T21:28:43.5709651Z Starting upload of usage-log-test-distributed-2-2-linux.2xlarge_10588877558.zip
2023-01-11T21:28:43.7468523Z Finished upload of usage-log-test-distributed-2-2-linux.2xlarge_10588877558.zip
2023-01-11T21:28:43.7624514Z ##[group]Run # shellcheck disable=SC2156
2023-01-11T21:28:43.7624968Z # shellcheck disable=SC2156
2023-01-11T21:28:43.7625283Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2023-01-11T21:28:43.7636449Z shell: /usr/bin/bash -e {0}
2023-01-11T21:28:43.7636634Z env:
2023-01-11T21:28:43.7636797Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:28:43.7637073Z   DOCKER_CONTAINER_ID: 16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58
2023-01-11T21:28:43.7637331Z ##[endgroup]
2023-01-11T21:28:45.6020270Z ##[group]Run set -x
2023-01-11T21:28:45.6020592Z set -x
2023-01-11T21:28:45.6020824Z python3 -m pip install -r requirements.txt
2023-01-11T21:28:45.6021061Z python3 -m pip install boto3==1.19.12
2023-01-11T21:28:45.6021355Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2023-01-11T21:28:45.6032699Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:28:45.6032905Z env:
2023-01-11T21:28:45.6033090Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:28:45.6033363Z   DOCKER_CONTAINER_ID: 16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58
2023-01-11T21:28:45.6033624Z   AWS_DEFAULT_REGION: us-east-1
2023-01-11T21:28:45.6033814Z   BRANCH: pull/91627
2023-01-11T21:28:45.6034000Z   TEST_CONFIG: distributed
2023-01-11T21:28:45.6034171Z   SHARD_NUMBER: 2
2023-01-11T21:28:45.6034383Z   BUILD_ENVIRONMENT: linux-focal-py3.7-gcc7
2023-01-11T21:28:45.6034595Z   PR_NUMBER: 91627
2023-01-11T21:28:45.6034805Z   PYTORCH_RETRY_TEST_CASES: 1
2023-01-11T21:28:45.6035011Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2023-01-11T21:28:45.6035230Z   SHA1: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:28:45.6035430Z   TAG: 
2023-01-11T21:28:45.6035594Z   WORKFLOW_ID: 3896099317
2023-01-11T21:28:45.6035912Z   GITHUB_TOKEN: ***
2023-01-11T21:28:45.6036101Z   GHA_WORKFLOW_JOB_ID: 10588877558
2023-01-11T21:28:45.6036285Z ##[endgroup]
2023-01-11T21:28:45.6060823Z + python3 -m pip install -r requirements.txt
2023-01-11T21:28:45.8153478Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T21:28:45.8447305Z Requirement already satisfied: astunparse in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (1.6.3)
2023-01-11T21:28:45.8473449Z Requirement already satisfied: expecttest in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (0.1.4)
2023-01-11T21:28:45.8481635Z Requirement already satisfied: future in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (0.18.2)
2023-01-11T21:28:45.8490127Z Requirement already satisfied: hypothesis in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (6.62.0)
2023-01-11T21:28:45.8847886Z Requirement already satisfied: numpy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (1.21.6)
2023-01-11T21:28:45.8857095Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1)
2023-01-11T21:28:45.8932061Z Requirement already satisfied: pyyaml in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (6.0)
2023-01-11T21:28:45.8940058Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0)
2023-01-11T21:28:45.9149142Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3)
2023-01-11T21:28:45.9311593Z Requirement already satisfied: six in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 11)) (1.16.0)
2023-01-11T21:28:45.9320183Z Requirement already satisfied: types-dataclasses in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 12)) (0.6.6)
2023-01-11T21:28:45.9326078Z Requirement already satisfied: typing_extensions in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 13)) (4.4.0)
2023-01-11T21:28:45.9335967Z Requirement already satisfied: sympy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 14)) (1.10.1)
2023-01-11T21:28:45.9354411Z Requirement already satisfied: filelock in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 15)) (3.9.0)
2023-01-11T21:28:45.9424142Z Requirement already satisfied: networkx in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 16)) (2.6.3)
2023-01-11T21:28:45.9577841Z Requirement already satisfied: jinja2 in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 17)) (3.1.2)
2023-01-11T21:28:45.9601930Z Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from astunparse->-r requirements.txt (line 2)) (0.38.4)
2023-01-11T21:28:45.9617838Z Requirement already satisfied: attrs>=19.2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (22.2.0)
2023-01-11T21:28:45.9865143Z Requirement already satisfied: exceptiongroup>=1.0.0; python_version < "3.11" in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (1.1.0)
2023-01-11T21:28:45.9882977Z Requirement already satisfied: sortedcontainers<3.0.0,>=2.1.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (2.4.0)
2023-01-11T21:28:45.9893502Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.14)
2023-01-11T21:28:46.0041882Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.12.7)
2023-01-11T21:28:46.0051069Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12)
2023-01-11T21:28:46.0069594Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4)
2023-01-11T21:28:46.0080746Z Requirement already satisfied: mpmath>=0.19 in /home/ec2-user/.local/lib/python3.7/site-packages (from sympy->-r requirements.txt (line 14)) (1.2.1)
2023-01-11T21:28:46.0134314Z Requirement already satisfied: MarkupSafe>=2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from jinja2->-r requirements.txt (line 17)) (2.1.1)
2023-01-11T21:28:46.1706337Z + python3 -m pip install boto3==1.19.12
2023-01-11T21:28:46.3763076Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T21:28:46.3948868Z Requirement already satisfied: boto3==1.19.12 in /home/ec2-user/.local/lib/python3.7/site-packages (1.19.12)
2023-01-11T21:28:46.3997324Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0)
2023-01-11T21:28:46.4010625Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (1.22.12)
2023-01-11T21:28:46.4061067Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.5.2)
2023-01-11T21:28:46.4086352Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.14)
2023-01-11T21:28:46.4238073Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2)
2023-01-11T21:28:46.4256998Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2023-01-11T21:28:46.7062817Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2023-01-11T21:31:59.6993474Z [scribe] Scribe access token not provided, sending report via boto3...
2023-01-11T21:31:59.6994211Z ERROR ENCOUNTERED WHEN UPLOADING TO SCRIBE: {"errorMessage":"2023-01-11T21:31:50.104Z 22d4bf6f-2142-4662-b73a-5a8379ec40c3 Task timed out after 60.00 seconds"}
2023-01-11T21:31:59.6994451Z 
2023-01-11T21:31:59.6995154Z ----- Historic stats comparison result ------
2023-01-11T21:31:59.6996329Z 
2023-01-11T21:31:59.6996571Z     job: linux-focal-py3.7-gcc7
2023-01-11T21:31:59.6997044Z     commit: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:31:59.6997323Z 
2023-01-11T21:31:59.6998113Z Commit graph (base is most recent master ancestor with at least one S3 report):
2023-01-11T21:31:59.6998442Z 
2023-01-11T21:31:59.6998583Z     : (master)
2023-01-11T21:31:59.6998846Z     |
2023-01-11T21:31:59.6999870Z     | * 8419ddda87 (HEAD)            total time  1692.96s
2023-01-11T21:31:59.7000283Z     | |
2023-01-11T21:31:59.7000585Z     | : (2 commits)
2023-01-11T21:31:59.7000875Z     |/
2023-01-11T21:31:59.7004842Z     * db2a237763 (base)   6 reports, total time  4537.52s ± 3598.58s
2023-01-11T21:31:59.7005245Z     * 2b0abd4ce3          6 reports, total time  4508.29s ± 3592.59s
2023-01-11T21:31:59.7005562Z     * f7939b21e1         18 reports, total time  2831.48s ± 3247.60s
2023-01-11T21:31:59.7005860Z     * cb3204823e          6 reports, total time  3677.58s ± 3486.86s
2023-01-11T21:31:59.7006222Z     * 6e236553f5          6 reports, total time  3702.23s ± 3507.16s
2023-01-11T21:31:59.7006533Z     * cce577b391          6 reports, total time  3768.82s ± 3640.72s
2023-01-11T21:31:59.7006831Z     * fae821c2f1          6 reports, total time  4476.74s ± 3618.76s
2023-01-11T21:31:59.7007141Z     * 0c3659586d          6 reports, total time  4420.82s ± 3571.24s
2023-01-11T21:31:59.7007444Z     * 122245985a          6 reports, total time  4479.72s ± 3631.94s
2023-01-11T21:31:59.7007748Z     * b797a24259          6 reports, total time  4511.06s ± 3679.54s
2023-01-11T21:31:59.7007938Z     |
2023-01-11T21:31:59.7008084Z     :
2023-01-11T21:31:59.7008181Z 
2023-01-11T21:31:59.7008298Z Removed  (across 1355 suites)     0 tests, totaling     0.00s
2023-01-11T21:31:59.7008541Z Modified (across    0 suites)     0 tests, totaling     0.00s
2023-01-11T21:31:59.7008795Z Added    (across  126 suites)  1194 tests, totaling +1692.96s
2023-01-11T21:31:59.7576944Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2023-01-11T21:31:59.7577191Z with:
2023-01-11T21:31:59.7577332Z env:
2023-01-11T21:31:59.7577503Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:31:59.7577780Z   DOCKER_CONTAINER_ID: 16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58
2023-01-11T21:31:59.7578024Z ##[endgroup]
2023-01-11T21:31:59.7592739Z ##[group]Run set -eou pipefail
2023-01-11T21:31:59.7592947Z set -eou pipefail
2023-01-11T21:31:59.7593125Z 
2023-01-11T21:31:59.7593358Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2023-01-11T21:31:59.7593605Z for _ in $(seq 1440); do
2023-01-11T21:31:59.7593822Z     # Break if no ssh session exists anymore
2023-01-11T21:31:59.7594035Z     if [ "$(who)" = "" ]; then
2023-01-11T21:31:59.7594214Z       break
2023-01-11T21:31:59.7594369Z     fi
2023-01-11T21:31:59.7594563Z     echo "."
2023-01-11T21:31:59.7594723Z     sleep 5
2023-01-11T21:31:59.7594889Z done
2023-01-11T21:31:59.7606198Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:31:59.7606402Z env:
2023-01-11T21:31:59.7606576Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:31:59.7606853Z   DOCKER_CONTAINER_ID: 16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58
2023-01-11T21:31:59.7607110Z ##[endgroup]
2023-01-11T21:31:59.7630863Z Holding runner for 2 hours until all ssh sessions have logged out
2023-01-11T21:31:59.7710821Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2023-01-11T21:31:59.7711214Z # ignore expansion of "docker ps -q" since it could be empty
2023-01-11T21:31:59.7711463Z # shellcheck disable=SC2046
2023-01-11T21:31:59.7711686Z docker stop $(docker ps -q) || true
2023-01-11T21:31:59.7711911Z # Prune all of the docker images
2023-01-11T21:31:59.7712112Z docker system prune -af
2023-01-11T21:31:59.7722647Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:31:59.7722866Z env:
2023-01-11T21:31:59.7723026Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:31:59.7723303Z   DOCKER_CONTAINER_ID: 16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58
2023-01-11T21:31:59.7723559Z ##[endgroup]
2023-01-11T21:32:00.2180314Z 16a6623252a2
2023-01-11T21:32:01.7038393Z Deleted Containers:
2023-01-11T21:32:01.7038908Z 16a6623252a2cc5b36bd21cb70dc58a762f2f588ce23450ae9d30ccf787a0d58
2023-01-11T21:32:01.7039128Z 
2023-01-11T21:32:07.9305416Z Deleted Images:
2023-01-11T21:32:07.9306593Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:32:07.9307355Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-py3.7-gcc7@sha256:d201d5d219c1681863b90b8d50e8b2382dd0a533dc69c772527c8255937144a4
2023-01-11T21:32:07.9307845Z deleted: sha256:872545647a28b88516eae4000db04fb3df455244b9807860ebbad7d21df7fbd3
2023-01-11T21:32:07.9308161Z deleted: sha256:0913ba780396d26b1a734b3dee7f02947640c13b56c904623d8d169cb2f445d9
2023-01-11T21:32:07.9308572Z deleted: sha256:2ab5e0d230803b5fd25b7b0f930eee8ab6c77cfac036eaa85c95a435b426df7e
2023-01-11T21:32:07.9308963Z deleted: sha256:058992102b05c2e3259d499eee2fb79686b842ff3b38268b218b9c9049bb4f4e
2023-01-11T21:32:07.9309331Z deleted: sha256:a3881c186a0b72148266b6499bd71e5a317b24fee69b6437d44f3c4975960695
2023-01-11T21:32:07.9309712Z deleted: sha256:13ab35328da38a4819bf04cd02facc248dfbaa82d5ec3a3b6e2c669dd985461b
2023-01-11T21:32:07.9310132Z deleted: sha256:aa2b1c0d1223bd8e06c4b3dd1f62593d69cd9d16cf4c46502020dd0ddabaedf4
2023-01-11T21:32:07.9310544Z deleted: sha256:07d56fb6f87c59cf211fb1d48faf0c0697dba1c8bd2eb68cfc32806d0660fb0b
2023-01-11T21:32:07.9310949Z deleted: sha256:b24802dbddd392269b952ab58802f68ad557e19cbce32bb194ab1c09e7eca6c5
2023-01-11T21:32:07.9314345Z deleted: sha256:ef000ff41666363f8fdeac09f9203e47a0ebaed82d61cd2aef4e648d75859d7c
2023-01-11T21:32:07.9315052Z deleted: sha256:b654f485842eed85c133b3d43fef43bfda9f7e718584e49590fa1ab94cbdb621
2023-01-11T21:32:07.9315556Z deleted: sha256:db407d4b58006712031749736db1c37874e5d4e908dfcba929dea8da63234d10
2023-01-11T21:32:07.9316042Z deleted: sha256:d93740e501b928fd7be659a8e1586a3d87e569c1544c2761ecdea7ab0997e7cc
2023-01-11T21:32:07.9316556Z deleted: sha256:ac19b11a34be7cb4598050f9aabd16ff0ff79831f485fb6000af0dfaddb436de
2023-01-11T21:32:07.9317052Z deleted: sha256:a5ae6c04d7731515ac814d89d2986f7910d0867922320cb22d751b3417011f43
2023-01-11T21:32:07.9317566Z deleted: sha256:6eb6bd31a4bd4d9a467b64ebdd39d775cba0ea1edb1de0b00ed3837d816425a8
2023-01-11T21:32:07.9318047Z deleted: sha256:19787711222d55795f879841531b67c7d4278c9f68c7c9510f023efc7e761faa
2023-01-11T21:32:07.9318522Z deleted: sha256:8fa2cc4e3117a1ec0a6d95bd9f17b254dbb0def34fcb15b3e7d6bb9a9f3e35c5
2023-01-11T21:32:07.9318989Z deleted: sha256:098023de3c889616f6862e6be3ec1c96adb5067276efcd2567245b9e39ea0fd9
2023-01-11T21:32:07.9319495Z deleted: sha256:2812d79bbc620bed5ab85a40ba81afc41f5c8b47dead2c7ae008f0dbf6a08d64
2023-01-11T21:32:07.9320018Z deleted: sha256:724cd734f88329c39fb2001c8b92ec22e9d9047f041df8d32978044c71884f31
2023-01-11T21:32:07.9320521Z deleted: sha256:8afdf6dfb1a843616a6386809d2ac8f394c29991357a9fde65af4382ebbe8ef2
2023-01-11T21:32:07.9321023Z deleted: sha256:ac10867395ba90a5662da28be64822423bde7ca92a50e646804c516a98e53004
2023-01-11T21:32:07.9321503Z deleted: sha256:d923c532b32b0d27a1fca08eb52224541269676efc35421ca1143f3bb64c0e7a
2023-01-11T21:32:07.9322011Z deleted: sha256:166fd4df8a671ab6d72685e787eb87bb17716ecfb8383732ce0330cc01de9366
2023-01-11T21:32:07.9322684Z deleted: sha256:0cc9007352917a0fe743d12ca9d2333b9123ddd5d501f9caee7281d837d5b287
2023-01-11T21:32:07.9323072Z deleted: sha256:ad1d657714a0c9feee756d8598c9ebdd7613bea5d3c53cf0a8ec89eb2c27ba2a
2023-01-11T21:32:07.9323407Z deleted: sha256:d4345474a810d8add867c09fddec1703c6f1ef96056681cbd404c3adf9c26231
2023-01-11T21:32:07.9323733Z deleted: sha256:90cfc4251be8e59835753e6518aefddf4f09951dde250ea0f3e10a09cd1bbef1
2023-01-11T21:32:07.9324040Z deleted: sha256:0f39cccb089ed0a8c80302e51e04579017d34266b2dbbf7bc09a826071de9da5
2023-01-11T21:32:07.9324341Z deleted: sha256:844c9188ba4b07d8d846104595214e63ec0c9818848540249393ced49196e702
2023-01-11T21:32:07.9324642Z deleted: sha256:b60329deafe4635066b5ce4cc72f1a3d11902a643666ffa7366b92b1e4c55f80
2023-01-11T21:32:07.9324966Z deleted: sha256:3545474c90ffd46a1ca93bbf91fa70bbbf7cdf7e0085a5e4e158164104c753ff
2023-01-11T21:32:07.9325265Z deleted: sha256:df43d5787137e8e2383f7d1627a5093edba3534745c272878d97083fdbf5f984
2023-01-11T21:32:07.9325588Z deleted: sha256:d9b003a92bbdfabe507ef0f8c00445a56c47fcf6f50e58ada5e86a6992f49c09
2023-01-11T21:32:07.9325926Z deleted: sha256:05d44c5f8014dfc2a6fb85fcaf86916e1305cda1571ee0f1e6320de59ce05bd7
2023-01-11T21:32:07.9326233Z deleted: sha256:530d276655c95d43dd64bb980673a2b13ebdb698a82a066a9b9fdf2e59dfe0bb
2023-01-11T21:32:07.9326547Z deleted: sha256:0daee3895574fbd6e661c964d60581233c5a8589b84d2590423deb3b29b51a46
2023-01-11T21:32:07.9326874Z deleted: sha256:8c76fd3e05eca362fe3d75f5a4c7133bd502ee43f59f17c64ed0c0c22fd46016
2023-01-11T21:32:07.9327192Z deleted: sha256:acb983c614c2804059e0970d18e5464a7542b4d551448270b8fd78cd782e6aa4
2023-01-11T21:32:07.9327499Z deleted: sha256:fda9d1b52f2ba36ac6b2b465a90f8982d9be82c45e4157ae554cde955389fcb3
2023-01-11T21:32:07.9327824Z deleted: sha256:1801c74cef4832dd65d4127c037ae1051a2a1eb90e7ce8120146851ace5946c7
2023-01-11T21:32:07.9328148Z deleted: sha256:240d82bb34e0f78bd9f1a0fc9663c2ddeb81020886c57c7188bb5b70b99e22ee
2023-01-11T21:32:07.9328454Z deleted: sha256:f88a7257109761d6e36e1e2de77e614d3de016b149db7113bcd8662c1fe99370
2023-01-11T21:32:07.9328762Z deleted: sha256:8a214e9e410e6400b71df4b1490fb8f2873203bc232689be875400d89f80b4c2
2023-01-11T21:32:07.9329071Z deleted: sha256:99332f27e027fff91df110db38b2f98a3b9c9212d130aef9a9ace15b4ca675a4
2023-01-11T21:32:07.9329396Z deleted: sha256:5e466b1d33a7b6f612f0d14153f80a4b2aaafd042765ec629a7099ed5d6b54a0
2023-01-11T21:32:07.9329780Z deleted: sha256:91eefd95450265ebde00db0268894212aeccff9e26fa2337760b1d75d20e4173
2023-01-11T21:32:07.9330111Z deleted: sha256:69463c6d56ebcccb292eff675b4d3f48de45fb09ecbb7847f9ccbe876d7e3504
2023-01-11T21:32:07.9330447Z deleted: sha256:296f50ac773cc9a4c23ba55c2564c926765f3c7d0a3fe6fe47e29aaa9ec9b22e
2023-01-11T21:32:07.9330757Z deleted: sha256:f89150a51cf7591d0e1a9f53ed7ea0c3aff94a467e80176a2d4f55c0e3bb41da
2023-01-11T21:32:07.9331315Z deleted: sha256:6d34afa8d6b4573ee5ab2c08738d797c78ad53a584f42d0ad9686080cb1cf955
2023-01-11T21:32:07.9331642Z deleted: sha256:77f6edeb37e215f3a81aea6d82d9c232700f550fb52211911ed569381c45de60
2023-01-11T21:32:07.9331958Z deleted: sha256:8744929d3f79f26f92321dfad103e20ab4d6649223c519635b8091c03e79bf23
2023-01-11T21:32:07.9332275Z deleted: sha256:cbcf68337e6007e98da7234219d686d9865e3691796e71ea82b931bcccbb3bcd
2023-01-11T21:32:07.9332745Z deleted: sha256:5ef8fcc78725941709d1712da344c0bc9f22450dde23c9da2c640dce6f58dc62
2023-01-11T21:32:07.9333165Z deleted: sha256:017f8ab4193b43fb8adcceda4b63e7a0e77f98591b0a0d8e2230cd43051d7651
2023-01-11T21:32:07.9333511Z deleted: sha256:b4c50db5a4dbcf2c4cf6b0e5c19878b2cd52e983fb5366ce83d7fd8e2539313a
2023-01-11T21:32:07.9333882Z deleted: sha256:0002c93bdb3704dd9e36ce5153ef637f84de253015f3ee330468dccdeacad60b
2023-01-11T21:32:07.9334060Z 
2023-01-11T21:32:07.9334160Z Total reclaimed space: 9.441GB
2023-01-11T21:32:07.9425441Z Post job cleanup.
2023-01-11T21:32:07.9474753Z Post job cleanup.
2023-01-11T21:32:08.0561500Z [command]/usr/bin/git version
2023-01-11T21:32:08.0606761Z git version 2.38.1
2023-01-11T21:32:08.0644597Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/2a07eda2-ce34-4e8a-a328-3bc20c710793' before making global git config changes
2023-01-11T21:32:08.0645421Z Adding repository directory to the temporary git global config as a safe directory
2023-01-11T21:32:08.0648833Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T21:32:08.0698446Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2023-01-11T21:32:08.0728403Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2023-01-11T21:32:08.1112703Z Entering 'android/libs/fbjni'
2023-01-11T21:32:08.1155882Z Entering 'third_party/FP16'
2023-01-11T21:32:08.1195515Z Entering 'third_party/FXdiv'
2023-01-11T21:32:08.1233004Z Entering 'third_party/NNPACK'
2023-01-11T21:32:08.1274016Z Entering 'third_party/QNNPACK'
2023-01-11T21:32:08.1314001Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T21:32:08.1348588Z Entering 'third_party/XNNPACK'
2023-01-11T21:32:08.1473497Z Entering 'third_party/benchmark'
2023-01-11T21:32:08.1518076Z Entering 'third_party/cpuinfo'
2023-01-11T21:32:08.1563578Z Entering 'third_party/cub'
2023-01-11T21:32:08.1596736Z Entering 'third_party/cudnn_frontend'
2023-01-11T21:32:08.1635562Z Entering 'third_party/cutlass'
2023-01-11T21:32:08.1675514Z Entering 'third_party/eigen'
2023-01-11T21:32:08.1711004Z Entering 'third_party/fbgemm'
2023-01-11T21:32:08.1745088Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:32:08.1778721Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:32:08.1812611Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:32:08.1845515Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:32:08.1880354Z Entering 'third_party/flatbuffers'
2023-01-11T21:32:08.1917215Z Entering 'third_party/fmt'
2023-01-11T21:32:08.1951698Z Entering 'third_party/foxi'
2023-01-11T21:32:08.1985701Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:32:08.2020131Z Entering 'third_party/gloo'
2023-01-11T21:32:08.2054207Z Entering 'third_party/googletest'
2023-01-11T21:32:08.2088196Z Entering 'third_party/ideep'
2023-01-11T21:32:08.2121311Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T21:32:08.2157751Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:32:08.2198557Z Entering 'third_party/ios-cmake'
2023-01-11T21:32:08.2233384Z Entering 'third_party/ittapi'
2023-01-11T21:32:08.2267036Z Entering 'third_party/kineto'
2023-01-11T21:32:08.2301064Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:32:08.2335153Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:32:08.2369659Z Entering 'third_party/nccl/nccl'
2023-01-11T21:32:08.2404438Z Entering 'third_party/neon2sse'
2023-01-11T21:32:08.2439214Z Entering 'third_party/nlohmann'
2023-01-11T21:32:08.2474496Z Entering 'third_party/onnx'
2023-01-11T21:32:08.2519589Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T21:32:08.2553352Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T21:32:08.2590521Z Entering 'third_party/onnx-tensorrt'
2023-01-11T21:32:08.2624107Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:32:08.2661491Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:32:08.2694317Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:32:08.2726730Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:32:08.2764037Z Entering 'third_party/pocketfft'
2023-01-11T21:32:08.2798170Z Entering 'third_party/protobuf'
2023-01-11T21:32:08.2836405Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:32:08.2871330Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T21:32:08.2906684Z Entering 'third_party/psimd'
2023-01-11T21:32:08.2941093Z Entering 'third_party/pthreadpool'
2023-01-11T21:32:08.2975555Z Entering 'third_party/pybind11'
2023-01-11T21:32:08.3009606Z Entering 'third_party/python-enum'
2023-01-11T21:32:08.3043352Z Entering 'third_party/python-peachpy'
2023-01-11T21:32:08.3076975Z Entering 'third_party/python-six'
2023-01-11T21:32:08.3110913Z Entering 'third_party/sleef'
2023-01-11T21:32:08.3145718Z Entering 'third_party/tbb'
2023-01-11T21:32:08.3183115Z Entering 'third_party/tensorpipe'
2023-01-11T21:32:08.3217380Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:32:08.3250427Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:32:08.3283507Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:32:08.3316015Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:32:08.3349852Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:32:08.3385420Z Entering 'third_party/zstd'
2023-01-11T21:32:08.3432524Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2023-01-11T21:32:08.3460575Z http.https://github.com/.extraheader
2023-01-11T21:32:08.3468415Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2023-01-11T21:32:08.3498904Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2023-01-11T21:32:08.3746068Z Entering 'android/libs/fbjni'
2023-01-11T21:32:08.3765877Z http.https://github.com/.extraheader
2023-01-11T21:32:08.3792923Z Entering 'third_party/FP16'
2023-01-11T21:32:08.3814417Z http.https://github.com/.extraheader
2023-01-11T21:32:08.3839809Z Entering 'third_party/FXdiv'
2023-01-11T21:32:08.3859700Z http.https://github.com/.extraheader
2023-01-11T21:32:08.3885995Z Entering 'third_party/NNPACK'
2023-01-11T21:32:08.3905761Z http.https://github.com/.extraheader
2023-01-11T21:32:08.3932430Z Entering 'third_party/QNNPACK'
2023-01-11T21:32:08.3952667Z http.https://github.com/.extraheader
2023-01-11T21:32:08.3979842Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T21:32:08.3999860Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4026917Z Entering 'third_party/XNNPACK'
2023-01-11T21:32:08.4046613Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4084510Z Entering 'third_party/benchmark'
2023-01-11T21:32:08.4104578Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4130463Z Entering 'third_party/cpuinfo'
2023-01-11T21:32:08.4149963Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4177122Z Entering 'third_party/cub'
2023-01-11T21:32:08.4196854Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4223500Z Entering 'third_party/cudnn_frontend'
2023-01-11T21:32:08.4242642Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4273517Z Entering 'third_party/cutlass'
2023-01-11T21:32:08.4293838Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4326776Z Entering 'third_party/eigen'
2023-01-11T21:32:08.4346039Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4374065Z Entering 'third_party/fbgemm'
2023-01-11T21:32:08.4393950Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4420002Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:32:08.4438977Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4464330Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:32:08.4483543Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4510587Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:32:08.4529684Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4555844Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:32:08.4575036Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4601505Z Entering 'third_party/flatbuffers'
2023-01-11T21:32:08.4621000Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4648751Z Entering 'third_party/fmt'
2023-01-11T21:32:08.4669203Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4695189Z Entering 'third_party/foxi'
2023-01-11T21:32:08.4715168Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4740210Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:32:08.4761032Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4787360Z Entering 'third_party/gloo'
2023-01-11T21:32:08.4807090Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4832950Z Entering 'third_party/googletest'
2023-01-11T21:32:08.4853433Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4879832Z Entering 'third_party/ideep'
2023-01-11T21:32:08.4899781Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4925804Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T21:32:08.4944915Z http.https://github.com/.extraheader
2023-01-11T21:32:08.4975074Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:32:08.4994581Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5037393Z Entering 'third_party/ios-cmake'
2023-01-11T21:32:08.5057812Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5083833Z Entering 'third_party/ittapi'
2023-01-11T21:32:08.5103471Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5129694Z Entering 'third_party/kineto'
2023-01-11T21:32:08.5150485Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5176435Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:32:08.5195922Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5222599Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:32:08.5242226Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5269681Z Entering 'third_party/nccl/nccl'
2023-01-11T21:32:08.5289535Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5315626Z Entering 'third_party/neon2sse'
2023-01-11T21:32:08.5336729Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5362863Z Entering 'third_party/nlohmann'
2023-01-11T21:32:08.5382428Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5410116Z Entering 'third_party/onnx'
2023-01-11T21:32:08.5430957Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5468899Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T21:32:08.5488422Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5525762Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T21:32:08.5545372Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5579051Z Entering 'third_party/onnx-tensorrt'
2023-01-11T21:32:08.5598552Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5623981Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:32:08.5642983Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5681078Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:32:08.5700606Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5726095Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:32:08.5746613Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5772221Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:32:08.5791758Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5821594Z Entering 'third_party/pocketfft'
2023-01-11T21:32:08.5840991Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5867084Z Entering 'third_party/protobuf'
2023-01-11T21:32:08.5886646Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5915750Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:32:08.5935592Z http.https://github.com/.extraheader
2023-01-11T21:32:08.5962107Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T21:32:08.5982164Z http.https://github.com/.extraheader
2023-01-11T21:32:08.6018476Z Entering 'third_party/psimd'
2023-01-11T21:32:08.6039583Z http.https://github.com/.extraheader
2023-01-11T21:32:08.6064992Z Entering 'third_party/pthreadpool'
2023-01-11T21:32:08.6085247Z http.https://github.com/.extraheader
2023-01-11T21:32:08.6111703Z Entering 'third_party/pybind11'
2023-01-11T21:32:08.6130623Z http.https://github.com/.extraheader
2023-01-11T21:32:08.6156448Z Entering 'third_party/python-enum'
2023-01-11T21:32:08.6176241Z http.https://github.com/.extraheader
2023-01-11T21:32:08.6201968Z Entering 'third_party/python-peachpy'
2023-01-11T21:32:08.6222209Z http.https://github.com/.extraheader
2023-01-11T21:32:08.6257143Z Entering 'third_party/python-six'
2023-01-11T21:32:08.6276739Z http.https://github.com/.extraheader
2023-01-11T21:32:08.6301948Z Entering 'third_party/sleef'
2023-01-11T21:32:08.6322113Z http.https://github.com/.extraheader
2023-01-11T21:32:08.6349094Z Entering 'third_party/tbb'
2023-01-11T21:32:08.6368922Z http.https://github.com/.extraheader
2023-01-11T21:32:08.6396029Z Entering 'third_party/tensorpipe'
2023-01-11T21:32:08.6416858Z http.https://github.com/.extraheader
2023-01-11T21:32:08.6443207Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:32:08.6462540Z http.https://github.com/.extraheader
2023-01-11T21:32:08.6488692Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:32:08.6509565Z http.https://github.com/.extraheader
2023-01-11T21:32:08.6534910Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:32:08.6553500Z http.https://github.com/.extraheader
2023-01-11T21:32:08.6578876Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:32:08.6597528Z http.https://github.com/.extraheader
2023-01-11T21:32:08.6622844Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:32:08.6642229Z http.https://github.com/.extraheader
2023-01-11T21:32:08.6671647Z Entering 'third_party/zstd'
2023-01-11T21:32:08.6691463Z http.https://github.com/.extraheader
2023-01-11T21:32:08.6961228Z Cleaning up orphan processes